2024-03-24T11:22:32.033 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T11:22:32.033 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924 --description orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} --verbose --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924/orig.config.yaml 2024-03-24T11:22:32.066 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:22:32.144 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924 branch: squid description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617924' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) mon_bind_msgr2: false sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 extra_packages: - cephadm selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAxsFIyqivlXUt6ACFkD4M+bW95KpstEbHQuXEvKg3MhB77DoW1P4FekH2ME4GHxcgvWaLTnIjzwhOkME/jJkno= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNY7Xz0JLlpCSUPwvmjpOZDtcny0WnTS/9xEOHSFotE3picEqPPWYAHn12VzT7Twb0OuHpWcLqfx3L4AoCjHWM4= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T11:22:32.144 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T11:22:32.144 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T11:22:32.145 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T11:22:32.146 INFO:teuthology.task.internal:Checking packages... 2024-03-24T11:22:32.170 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T11:22:32.170 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:22:32.170 INFO:teuthology.packaging:ref: None 2024-03-24T11:22:32.170 INFO:teuthology.packaging:tag: None 2024-03-24T11:22:32.171 INFO:teuthology.packaging:branch: squid 2024-03-24T11:22:32.171 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:22:32.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T11:22:32.413 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T11:22:32.415 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T11:22:32.422 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T11:22:32.422 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T11:22:32.446 INFO:teuthology.task.internal:Saving configuration 2024-03-24T11:22:32.458 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T11:22:32.464 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:22:32.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:17:24.167511', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAxsFIyqivlXUt6ACFkD4M+bW95KpstEbHQuXEvKg3MhB77DoW1P4FekH2ME4GHxcgvWaLTnIjzwhOkME/jJkno='} 2024-03-24T11:22:32.514 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:17:24.165868', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNY7Xz0JLlpCSUPwvmjpOZDtcny0WnTS/9xEOHSFotE3picEqPPWYAHn12VzT7Twb0OuHpWcLqfx3L4AoCjHWM4='} 2024-03-24T11:22:32.515 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T11:22:32.521 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T11:22:32.521 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-24T11:22:32.522 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T11:22:32.593 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbad6cff0d0>, signals=[15]) 2024-03-24T11:22:32.593 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T11:22:32.600 INFO:teuthology.task.internal:Opening connections... 2024-03-24T11:22:32.600 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-03-24T11:22:32.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:32.693 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-03-24T11:22:32.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:32.785 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T11:22:32.791 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-03-24T11:22:32.808 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-03-24T11:22:32.808 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-03-24T11:22:32.865 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:22:32.866 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:22:32.867 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-03-24T11:22:32.897 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-03-24T11:22:32.915 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-03-24T11:22:32.915 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-03-24T11:22:32.971 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-03-24T11:22:32.971 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-03-24T11:22:32.971 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:22:32.972 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:22:32.973 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-03-24T11:22:32.997 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T11:22:33.006 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T11:22:33.013 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T11:22:33.013 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:22:33.016 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:22:33.032 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T11:22:33.039 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T11:22:33.039 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:22:33.073 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:22:33.287 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T11:22:33.344 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T11:22:33.363 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T11:22:33.363 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T11:22:33.364 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:22:33.364 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:22:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.381 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-24T11:22:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.382 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-24T11:22:33.437 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:22:33.437 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-361.el9.x86_64 2024-03-24T11:22:33.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-03-24T11:22:33.439 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:22:33.439 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.14.0-361.el9.x86_64 2024-03-24T11:22:33.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-03-24T11:22:35.639 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T11:22:35.943 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 7.7 MB/s | 6.7 MB 00:00 2024-03-24T11:22:37.023 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-24T11:22:37.318 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-24T11:22:38.205 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.0 MB 00:00 2024-03-24T11:22:40.215 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-24T11:22:41.167 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-24T11:22:42.968 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-24T11:22:51.798 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 769 kB/s | 6.1 MB 00:08 2024-03-24T11:22:53.240 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-24T11:22:54.644 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 764 kB/s | 6.1 MB 00:08 2024-03-24T11:22:55.450 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-03-24T11:22:56.127 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-24T11:22:57.812 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-24T11:23:00.358 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-24T11:23:02.456 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:23:02.457 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:23:02.546 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:23:02.556 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:23:02.556 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-03-24T11:23:02.556 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:23:02.556 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:23:02.556 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:23:02.556 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-03-24T11:23:02.557 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:23:02.558 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 76 M 2024-03-24T11:23:02.558 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 123 M 2024-03-24T11:23:02.558 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:23:02.812 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-24T11:23:03.051 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 193 kB/s | 44 kB 00:00 2024-03-24T11:23:03.476 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-24T11:23:04.043 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 28 MB/s | 27 MB 00:00 2024-03-24T11:23:04.668 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 18 MB/s | 33 MB 00:01 2024-03-24T11:23:04.671 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:23:04.671 INFO:teuthology.orchestra.run.smithi119.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-24T11:23:04.927 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:23:04.927 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:23:05.032 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:23:05.042 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:23:05.042 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-24T11:23:05.042 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-03-24T11:23:05.043 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:23:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 76 M 2024-03-24T11:23:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-03-24T11:23:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:23:05.322 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 344 kB/s | 44 kB 00:00 2024-03-24T11:23:05.343 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:23:06.694 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:23:06.695 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:23:09.153 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:23:09.774 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:23:09.862 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:23:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:23:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:23:17.422 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:23:17.548 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:21.587 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 931 kB/s | 15 MB 00:16 2024-03-24T11:23:22.292 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:31.478 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.1 MB/s | 27 MB 00:26 2024-03-24T11:23:35.053 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.1 MB/s | 33 MB 00:29 2024-03-24T11:23:35.053 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:23:35.053 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.5 MB/s | 76 MB 00:30 2024-03-24T11:23:35.721 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:23:36.080 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:23:36.080 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:23:37.110 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:23:37.111 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:23:39.564 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:23:40.165 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:23:40.252 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:23:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:23:43.104 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:23:47.790 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:23:47.912 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:51.719 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:51.719 INFO:teuthology.orchestra.run.smithi119.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:23:51.719 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:23:51.762 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:52.153 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:52.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:23:52.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:23:52.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:23:52.532 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:52.532 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:23:52.532 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:23:52.532 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:23:52.532 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:23:52.533 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:23:52.533 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:23:52.533 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:23:52.533 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:23:52.666 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:23:52.873 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:23:52.978 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:23:52.979 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:23:52.979 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:23:52.979 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:23:52.979 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:23:52.979 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T11:23:52.979 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:23:52.979 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:23:53.059 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:23:53.060 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:23:53.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-03-24T11:23:53.104 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-03-24T11:23:53.104 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 68.el9 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8264254 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:23:53.105 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:23:53.106 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:23:53.107 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:23:53.107 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:23:53.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:23:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-03-24T11:23:55.031 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:23:55.060 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-03-24T11:23:55.068 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-03-24T11:23:55.084 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.f3ITp5hTuW 2024-03-24T11:23:55.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.f3ITp5hTuW 2024-03-24T11:23:55.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.f3ITp5hTuW 2024-03-24T11:23:55.288 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.f3ITp5hTuW is 6KB 2024-03-24T11:23:55.339 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.f3ITp5hTuW 2024-03-24T11:23:55.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:23:55.422 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:23:55.422 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:23:55.422 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:23:55.422 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:23:55.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:23:55.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-03-24T11:24:21.892 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:24:21.892 INFO:teuthology.orchestra.run.smithi114.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:24:21.892 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:24:21.936 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:24:22.323 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:24:22.323 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:24:22.324 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:24:22.324 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:24:22.719 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:24:22.720 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:24:23.059 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:24:23.162 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:24:23.162 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:24:23.162 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:24:23.162 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:24:23.162 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:24:23.163 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T11:24:23.163 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:24:23.163 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:24:23.242 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:24:23.242 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:24:23.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-03-24T11:24:23.276 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:24:23.277 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:24:23.278 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:24:23.279 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:24:23.279 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:24:23.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:24:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-03-24T11:24:25.292 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:24:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-03-24T11:24:25.330 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-24T11:24:25.344 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.n7ECkYhTXW 2024-03-24T11:24:25.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.n7ECkYhTXW 2024-03-24T11:24:25.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.n7ECkYhTXW 2024-03-24T11:24:25.552 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.n7ECkYhTXW is 6KB 2024-03-24T11:24:25.583 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T11:24:25.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-24T11:24:25.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:25.603 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.n7ECkYhTXW 2024-03-24T11:24:25.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:24:25.689 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:24:25.689 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:24:25.689 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:24:25.689 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:24:25.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:24:25.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-03-24T11:24:43.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-24T11:24:52.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-24T11:24:52.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:55.842 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T11:24:55.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-24T11:24:55.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:55.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-24T11:25:08.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-24T11:25:08.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:25:14.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-03-24T11:25:23.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-24T11:25:23.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:25:26.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-03-24T11:25:38.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-24T11:25:38.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:25:39.766 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-24T11:25:40.132 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-24T11:25:40.133 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:25:40.133 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-24T11:25:40.190 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:25:40.191 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:25:40.191 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:25:40.191 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:25:41.191 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T11:25:41.192 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:25:41.271 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-03-24T11:25:41.309 DEBUG:teuthology.parallel:result is None 2024-03-24T11:25:56.895 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-03-24T11:26:11.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-24T11:26:11.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:26:12.323 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-24T11:26:12.775 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-24T11:26:12.776 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:26:12.776 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-24T11:26:12.789 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:26:12.790 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:26:12.790 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:26:12.790 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:26:13.793 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T11:26:13.793 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:26:13.857 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-03-24T11:26:13.906 DEBUG:teuthology.parallel:result is None 2024-03-24T11:26:13.907 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:26:13.914 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:26:13.915 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:26:13.918 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:26:13.936 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:26:13.944 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:26:13.954 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:26:13.954 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:26:13.973 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:26:14.009 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:26:14.016 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:26:14.017 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:26:14.036 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:26:14.068 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:26:14.170 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:26:14.170 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:26:14.170 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:26:14.170 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:26:14.189 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:26:14.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:26:14.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:26:14.244 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:26:14.254 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:26:14.255 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:26:14.282 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:26:14.301 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:26:14.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:26:14.434 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:26:14.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:26:14.571 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:26:14.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:26:14.648 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:26:14.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:26:14.770 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:26:14.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:26:14.870 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:26:14.870 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:26:14.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-03-24T11:26:14.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-03-24T11:26:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:26:15.042 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:26:15.343 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:26:15.791 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:26:15.792 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:26:16.115 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:26:16.320 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T11:26:16.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-03-24T11:26:16.430 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-03-24T11:26:16.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-03-24T11:26:16.492 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-03-24T11:26:16.494 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:26:16.494 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-03-24T11:26:16.518 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-03-24T11:26:16.519 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-03-24T11:26:16.539 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-03-24T11:26:16.539 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:26:16.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:26:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:26:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.612 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:26:16.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:26:16.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:26:16.691 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:26:16.705 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:26:16.731 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:26:16.732 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:30:26.125 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2024-03-24T11:30:26.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-24T11:30:26.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:30:26.218 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-24T11:30:26.297 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-24T11:30:26.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-24T11:30:26.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:30:26.386 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-24T11:30:26.464 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-24T11:30:26.465 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:30:26.475 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:30:26.475 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:30:26.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:30:26.478 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:30:26.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:30:26.507 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:30:26.523 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:30:26.536 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:30:26.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-03-24T11:30:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:30:26.566 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-03-24T11:30:26.582 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-03-24T11:30:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:30:26.596 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-03-24T11:30:26.598 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:30:26.612 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:30:26.629 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:30:26.646 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:30:26.649 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:30:26.649 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-24T11:30:26.650 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.650 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.650 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.650 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.692 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:30:26.696 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:30:26.696 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-24T11:30:26.696 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.696 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.696 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.696 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:30:26.696 INFO:teuthology.run_tasks:Running task pexec... 2024-03-24T11:30:26.705 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-24T11:30:26.705 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi114.front.sepia.ceph.com 2024-03-24T11:30:26.706 DEBUG:teuthology.orchestra.run.smithi114:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:30:26.706 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi119.front.sepia.ceph.com 2024-03-24T11:30:26.706 DEBUG:teuthology.orchestra.run.smithi119:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:30:27.282 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:48 ago on Sun 24 Mar 2024 11:29:39 AM UTC. 2024-03-24T11:30:27.316 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:47 ago on Sun 24 Mar 2024 11:29:40 AM UTC. 2024-03-24T11:30:27.428 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:30:27.439 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:30:27.439 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-24T11:30:27.439 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:30:27.440 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:30:27.440 INFO:teuthology.orchestra.run.smithi114.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:30:27.440 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:30:27.440 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:30:27.440 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:30:27.440 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:30:27.440 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:30:27.442 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.1 M 2024-03-24T11:30:27.442 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 9.8 M 2024-03-24T11:30:27.443 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:30:27.460 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:30:27.474 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:30:27.475 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:30:27.484 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.1 M 2024-03-24T11:30:27.484 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 9.8 M 2024-03-24T11:30:27.484 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:30:27.704 INFO:teuthology.orchestra.run.smithi119.stdout:runc-1.1.12-2.el9.x86_64.rpm 19 MB/s | 3.1 MB 00:00 2024-03-24T11:30:27.705 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:30:27.705 INFO:teuthology.orchestra.run.smithi119.stdout:Total 14 MB/s | 3.1 MB 00:00 2024-03-24T11:30:27.740 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:30:27.746 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:30:27.746 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:30:27.824 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:30:27.824 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:30:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:30:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:30:28.675 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:30:29.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:30:29.005 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:30:29.005 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:30:29.005 INFO:teuthology.orchestra.run.smithi119.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:30:29.005 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:30:29.005 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:30:29.277 DEBUG:teuthology.parallel:result is None 2024-03-24T11:30:34.477 INFO:teuthology.orchestra.run.smithi114.stdout:runc-1.1.12-2.el9.x86_64.rpm 468 kB/s | 3.1 MB 00:06 2024-03-24T11:30:34.477 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:30:34.477 INFO:teuthology.orchestra.run.smithi114.stdout:Total 455 kB/s | 3.1 MB 00:07 2024-03-24T11:30:34.514 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:30:34.520 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:30:34.520 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:30:34.603 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:30:34.603 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:30:34.828 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:30:34.920 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:30:35.382 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:30:35.684 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:30:35.684 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:30:35.684 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:30:35.685 INFO:teuthology.orchestra.run.smithi114.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:30:35.685 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:30:35.685 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:30:35.939 DEBUG:teuthology.parallel:result is None 2024-03-24T11:30:35.940 INFO:teuthology.run_tasks:Running task install... 2024-03-24T11:30:35.949 DEBUG:teuthology.task.install:project ceph 2024-03-24T11:30:35.949 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}, 'extra_packages': ['cephadm']} 2024-03-24T11:30:35.949 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:30:35.949 INFO:teuthology.task.install:Using flavor: default 2024-03-24T11:30:35.955 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T11:30:35.955 INFO:teuthology.task.install:extra packages: [] 2024-03-24T11:30:35.955 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:30:35.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:30:35.957 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:30:35.957 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:30:36.104 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T11:30:36.104 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T11:30:36.323 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T11:30:36.323 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T11:30:36.371 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:30:36.372 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:30:36.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:30:36.405 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:30:36.406 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:30:36.432 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:30:36.433 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:30:36.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:30:36.460 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:30:36.461 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:30:36.477 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:30:36.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:30:36.560 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:30:36.609 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:30:36.628 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-03-24T11:30:36.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-03-24T11:30:36.675 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-03-24T11:30:36.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-03-24T11:30:36.875 INFO:teuthology.orchestra.run.smithi119.stdout:74 files removed 2024-03-24T11:30:36.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-03-24T11:30:36.915 INFO:teuthology.orchestra.run.smithi114.stdout:74 files removed 2024-03-24T11:30:36.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-03-24T11:30:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 252 kB/s | 82 kB 00:00 2024-03-24T11:30:37.568 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 223 kB/s | 82 kB 00:00 2024-03-24T11:30:37.994 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-24T11:30:38.000 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-24T11:30:38.336 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-24T11:30:38.350 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-24T11:30:38.747 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T11:30:39.028 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-24T11:30:40.420 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-24T11:30:40.472 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-24T11:30:41.971 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-03-24T11:30:42.027 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-03-24T11:30:43.603 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-03-24T11:30:43.880 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T11:30:48.340 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-24T11:30:49.941 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-24T11:30:51.250 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-24T11:30:54.903 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 799 kB/s | 6.1 MB 00:07 2024-03-24T11:30:56.217 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-24T11:30:56.554 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-24T11:30:57.952 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-24T11:30:58.706 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:30:58.708 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T11:30:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T11:30:58.710 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout:Install 30 Packages 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-03-24T11:30:58.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:30:58.712 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 58 M 2024-03-24T11:30:58.712 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:30:59.585 INFO:teuthology.orchestra.run.smithi119.stdout:(1/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x86 8.1 MB/s | 5.3 MB 00:00 2024-03-24T11:30:59.626 INFO:teuthology.orchestra.run.smithi119.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 605 kB/s | 25 kB 00:00 2024-03-24T11:30:59.710 INFO:teuthology.orchestra.run.smithi119.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-24T11:30:59.785 INFO:teuthology.orchestra.run.smithi119.stdout:(4/32): libradosstriper1-19.0.0-1667.gdb0330b1. 6.5 MB/s | 498 kB 00:00 2024-03-24T11:31:00.211 INFO:teuthology.orchestra.run.smithi119.stdout:(5/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_6 12 MB/s | 5.1 MB 00:00 2024-03-24T11:31:00.261 INFO:teuthology.orchestra.run.smithi119.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 909 kB/s | 45 kB 00:00 2024-03-24T11:31:00.336 INFO:teuthology.orchestra.run.smithi119.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 1.7 MB/s | 131 kB 00:00 2024-03-24T11:31:00.395 INFO:teuthology.orchestra.run.smithi119.stdout:(8/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 2.7 MB/s | 161 kB 00:00 2024-03-24T11:31:00.462 INFO:teuthology.orchestra.run.smithi119.stdout:(9/32): python3-rados-19.0.0-1667.gdb0330b1.el9 4.7 MB/s | 320 kB 00:00 2024-03-24T11:31:00.520 INFO:teuthology.orchestra.run.smithi119.stdout:(10/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 5.0 MB/s | 298 kB 00:00 2024-03-24T11:31:00.579 INFO:teuthology.orchestra.run.smithi119.stdout:(11/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.7 MB/s | 100 kB 00:00 2024-03-24T11:31:00.729 INFO:teuthology.orchestra.run.smithi119.stdout:(12/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9 5.4 MB/s | 9.8 MB 00:01 2024-03-24T11:31:00.788 INFO:teuthology.orchestra.run.smithi119.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 507 kB/s | 106 kB 00:00 2024-03-24T11:31:00.813 INFO:teuthology.orchestra.run.smithi119.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-24T11:31:00.830 INFO:teuthology.orchestra.run.smithi119.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-24T11:31:00.872 INFO:teuthology.orchestra.run.smithi119.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-24T11:31:00.897 INFO:teuthology.orchestra.run.smithi119.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-24T11:31:00.923 INFO:teuthology.orchestra.run.smithi119.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 819 kB/s | 41 kB 00:00 2024-03-24T11:31:01.039 INFO:teuthology.orchestra.run.smithi119.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 299 kB/s | 35 kB 00:00 2024-03-24T11:31:01.098 INFO:teuthology.orchestra.run.smithi119.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 762 kB/s | 44 kB 00:00 2024-03-24T11:31:01.290 INFO:teuthology.orchestra.run.smithi119.stdout:(21/32): ceph-common-19.0.0-1667.gdb0330b1.el9. 8.4 MB/s | 20 MB 00:02 2024-03-24T11:31:01.315 INFO:teuthology.orchestra.run.smithi119.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 518 kB/s | 216 kB 00:00 2024-03-24T11:31:01.441 INFO:teuthology.orchestra.run.smithi119.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-03-24T11:31:01.508 INFO:teuthology.orchestra.run.smithi119.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 755 kB/s | 309 kB 00:00 2024-03-24T11:31:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-24T11:31:01.583 INFO:teuthology.orchestra.run.smithi119.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-24T11:31:01.667 INFO:teuthology.orchestra.run.smithi119.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-24T11:31:01.850 INFO:teuthology.orchestra.run.smithi119.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-03-24T11:31:01.934 INFO:teuthology.orchestra.run.smithi119.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-03-24T11:31:01.984 INFO:teuthology.orchestra.run.smithi119.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-03-24T11:31:02.226 INFO:teuthology.orchestra.run.smithi119.stdout:(31/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 9.0 MB/s | 3.4 MB 00:00 2024-03-24T11:31:02.310 INFO:teuthology.orchestra.run.smithi119.stdout:(32/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 8.3 MB/s | 3.1 MB 00:00 2024-03-24T11:31:02.312 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:02.312 INFO:teuthology.orchestra.run.smithi119.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-24T11:31:02.345 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:31:02.383 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:31:02.383 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:31:02.869 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:31:02.870 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:31:02.882 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-24T11:31:04.280 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:31:04.367 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T11:31:04.398 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T11:31:04.423 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T11:31:04.446 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:31:04.461 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:31:04.646 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:31:04.701 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:31:05.079 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T11:31:05.108 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:31:05.200 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:31:05.290 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T11:31:05.326 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:31:05.328 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T11:31:05.329 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:31:05.330 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-03-24T11:31:05.331 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T11:31:05.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:05.332 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:31:05.332 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:05.332 INFO:teuthology.orchestra.run.smithi114.stdout:Install 30 Packages 2024-03-24T11:31:05.332 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-03-24T11:31:05.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:05.332 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 58 M 2024-03-24T11:31:05.333 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:31:05.335 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:31:05.406 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:31:05.435 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T11:31:05.479 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T11:31:05.548 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:31:05.571 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:31:05.625 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:31:05.652 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T11:31:05.763 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T11:31:05.804 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T11:31:05.844 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T11:31:05.877 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T11:31:05.906 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T11:31:05.938 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T11:31:06.200 INFO:teuthology.orchestra.run.smithi114.stdout:(1/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x86 8.3 MB/s | 5.3 MB 00:00 2024-03-24T11:31:06.242 INFO:teuthology.orchestra.run.smithi114.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 607 kB/s | 25 kB 00:00 2024-03-24T11:31:06.260 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T11:31:06.294 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:31:06.309 INFO:teuthology.orchestra.run.smithi114.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 11 MB/s | 742 kB 00:00 2024-03-24T11:31:06.359 INFO:teuthology.orchestra.run.smithi114.stdout:(4/32): libradosstriper1-19.0.0-1667.gdb0330b1. 9.8 MB/s | 498 kB 00:00 2024-03-24T11:31:06.366 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:31:06.398 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T11:31:06.438 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T11:31:06.643 INFO:teuthology.orchestra.run.smithi114.stdout:(5/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_6 18 MB/s | 5.1 MB 00:00 2024-03-24T11:31:06.685 INFO:teuthology.orchestra.run.smithi114.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 1.1 MB/s | 45 kB 00:00 2024-03-24T11:31:06.727 INFO:teuthology.orchestra.run.smithi114.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 3.1 MB/s | 131 kB 00:00 2024-03-24T11:31:06.777 INFO:teuthology.orchestra.run.smithi114.stdout:(8/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 3.1 MB/s | 161 kB 00:00 2024-03-24T11:31:06.836 INFO:teuthology.orchestra.run.smithi114.stdout:(9/32): python3-rados-19.0.0-1667.gdb0330b1.el9 5.4 MB/s | 320 kB 00:00 2024-03-24T11:31:06.889 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T11:31:06.894 INFO:teuthology.orchestra.run.smithi114.stdout:(10/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 5.0 MB/s | 298 kB 00:00 2024-03-24T11:31:06.913 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:31:06.944 INFO:teuthology.orchestra.run.smithi114.stdout:(11/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.9 MB/s | 100 kB 00:00 2024-03-24T11:31:06.967 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:31:07.034 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T11:31:07.070 INFO:teuthology.orchestra.run.smithi114.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 846 kB/s | 106 kB 00:00 2024-03-24T11:31:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T11:31:07.128 INFO:teuthology.orchestra.run.smithi114.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-24T11:31:07.170 INFO:teuthology.orchestra.run.smithi114.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-24T11:31:07.271 INFO:teuthology.orchestra.run.smithi114.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-24T11:31:07.329 INFO:teuthology.orchestra.run.smithi114.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-24T11:31:07.380 INFO:teuthology.orchestra.run.smithi114.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 823 kB/s | 41 kB 00:00 2024-03-24T11:31:07.672 INFO:teuthology.orchestra.run.smithi114.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 742 kB/s | 216 kB 00:00 2024-03-24T11:31:07.722 INFO:teuthology.orchestra.run.smithi114.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-24T11:31:07.772 INFO:teuthology.orchestra.run.smithi114.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 892 kB/s | 44 kB 00:00 2024-03-24T11:31:08.081 INFO:teuthology.orchestra.run.smithi114.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-24T11:31:08.181 INFO:teuthology.orchestra.run.smithi114.stdout:(22/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9 3.7 MB/s | 9.8 MB 00:02 2024-03-24T11:31:08.390 INFO:teuthology.orchestra.run.smithi114.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-24T11:31:08.415 INFO:teuthology.orchestra.run.smithi114.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 111 kB/s | 26 kB 00:00 2024-03-24T11:31:08.457 INFO:teuthology.orchestra.run.smithi114.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-24T11:31:08.541 INFO:teuthology.orchestra.run.smithi114.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-24T11:31:08.566 INFO:teuthology.orchestra.run.smithi114.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 447 kB/s | 67 kB 00:00 2024-03-24T11:31:08.758 INFO:teuthology.orchestra.run.smithi114.stdout:(28/32): ceph-common-19.0.0-1667.gdb0330b1.el9. 6.2 MB/s | 20 MB 00:03 2024-03-24T11:31:08.783 INFO:teuthology.orchestra.run.smithi114.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 791 kB/s | 192 kB 00:00 2024-03-24T11:31:09.000 INFO:teuthology.orchestra.run.smithi114.stdout:(30/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 14 MB/s | 3.1 MB 00:00 2024-03-24T11:31:09.092 INFO:teuthology.orchestra.run.smithi114.stdout:(31/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-24T11:31:09.216 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:31:09.262 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:31:09.317 INFO:teuthology.orchestra.run.smithi114.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-24T11:31:09.320 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:09.320 INFO:teuthology.orchestra.run.smithi114.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-24T11:31:09.351 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:31:09.386 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:31:09.387 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:31:09.829 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:31:09.865 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:31:09.875 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:31:09.876 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:31:09.940 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:31:09.940 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:31:09.940 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:31:09.940 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:09.975 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:31:11.306 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:31:11.390 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T11:31:11.430 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T11:31:11.456 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T11:31:11.479 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:31:11.494 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:31:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:31:11.726 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:31:12.043 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T11:31:12.099 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:31:12.190 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:31:12.279 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T11:31:12.317 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:31:12.364 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:31:12.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T11:31:12.445 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T11:31:12.512 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:31:12.537 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:31:12.599 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:31:12.635 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T11:31:12.744 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T11:31:12.786 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T11:31:12.846 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T11:31:12.876 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T11:31:12.905 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T11:31:12.937 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T11:31:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T11:31:13.276 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:31:13.349 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:31:13.381 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T11:31:13.419 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T11:31:13.866 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T11:31:13.895 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:31:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:31:14.033 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T11:31:14.064 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T11:31:16.220 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:31:16.261 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:31:16.821 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:31:16.856 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:31:16.952 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:31:16.952 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:31:16.952 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:31:16.952 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:16.999 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:31:19.508 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:31:19.508 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-03-24T11:31:19.508 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-03-24T11:31:19.508 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-03-24T11:31:19.509 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-03-24T11:31:19.509 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-03-24T11:31:19.509 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-03-24T11:31:19.509 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-03-24T11:31:19.509 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:20.386 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:31:20.410 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:31:20.410 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:20.411 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:31:20.411 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:31:20.411 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:31:20.411 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:20.412 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:31:20.488 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:31:20.488 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:31:21.632 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:31:21.632 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T11:31:21.632 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T11:31:21.632 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T11:31:21.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T11:31:21.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T11:31:21.635 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T11:31:21.635 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T11:31:21.635 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:31:21.635 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T11:31:21.635 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T11:31:22.565 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.566 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.567 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:22.568 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:22.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-03-24T11:31:23.394 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:27 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:31:23.548 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:31:23.549 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:23.549 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:31:23.549 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:23.549 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:31:23.549 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-03-24T11:31:23.550 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:23.551 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 48 M 2024-03-24T11:31:23.551 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 205 M 2024-03-24T11:31:23.551 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:31:23.985 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-24T11:31:24.052 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 691 kB/s | 163 kB 00:00 2024-03-24T11:31:24.077 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-24T11:31:24.094 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-24T11:31:24.319 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-03-24T11:31:26.370 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-24T11:31:26.373 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:26.374 INFO:teuthology.orchestra.run.smithi119.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-24T11:31:26.388 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:31:26.408 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:31:26.408 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:31:26.682 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:31:26.682 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-03-24T11:31:26.682 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-03-24T11:31:26.682 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-03-24T11:31:26.682 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-03-24T11:31:26.683 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-03-24T11:31:26.683 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-03-24T11:31:26.683 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-03-24T11:31:26.683 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:26.796 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:31:26.797 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:31:27.384 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:31:27.454 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:31:27.497 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:31:27.537 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:31:27.579 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:31:27.599 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:31:27.606 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:31:27.623 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:31:27.623 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:27.624 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:31:27.624 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:31:27.624 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:31:27.624 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:27.625 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:31:27.727 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:31:27.727 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:31:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:31:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T11:31:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T11:31:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T11:31:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T11:31:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T11:31:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T11:31:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T11:31:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T11:31:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T11:31:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:31:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T11:31:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T11:31:30.014 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.015 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T11:31:30.016 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T11:31:30.017 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:31:30.017 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:30.017 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:31:30.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-03-24T11:31:30.925 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:28 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:31:31.072 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:31:31.073 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:31.073 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:31:31.073 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:31.073 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:31:31.073 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-03-24T11:31:31.074 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 48 M 2024-03-24T11:31:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 205 M 2024-03-24T11:31:31.075 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:31:31.524 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-24T11:31:31.591 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-24T11:31:31.616 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 611 kB/s | 163 kB 00:00 2024-03-24T11:31:31.633 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-24T11:31:31.808 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 976 kB/s | 187 kB 00:00 2024-03-24T11:31:32.128 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:31:32.197 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:31:34.467 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 15 MB/s | 47 MB 00:03 2024-03-24T11:31:34.469 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:34.470 INFO:teuthology.orchestra.run.smithi114.stdout:Total 14 MB/s | 48 MB 00:03 2024-03-24T11:31:34.484 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:31:34.505 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:31:34.506 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:31:34.583 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:31:34.583 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T11:31:34.583 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T11:31:34.583 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:31:34.583 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:31:34.583 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:31:34.889 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:31:34.890 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:35.033 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:35.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-03-24T11:31:35.462 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:31:35.543 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:31:35.595 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:31:35.618 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:31:35.659 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:31:35.687 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:31:35.875 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:39 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:31:36.042 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:31:36.044 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-03-24T11:31:36.044 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:31:36.045 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:31:36.046 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:31:36.047 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-03-24T11:31:36.048 INFO:teuthology.orchestra.run.smithi119.stdout:Install 36 Packages 2024-03-24T11:31:36.048 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:36.048 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 30 M 2024-03-24T11:31:36.048 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 107 M 2024-03-24T11:31:36.048 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:31:36.822 INFO:teuthology.orchestra.run.smithi119.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-24T11:31:37.047 INFO:teuthology.orchestra.run.smithi119.stdout:(2/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2024-03-24T11:31:37.097 INFO:teuthology.orchestra.run.smithi119.stdout:(3/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-03-24T11:31:37.190 INFO:teuthology.orchestra.run.smithi119.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 2.6 MB/s | 247 kB 00:00 2024-03-24T11:31:37.306 INFO:teuthology.orchestra.run.smithi119.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-24T11:31:37.349 INFO:teuthology.orchestra.run.smithi119.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T11:31:37.373 INFO:teuthology.orchestra.run.smithi119.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-24T11:31:37.399 INFO:teuthology.orchestra.run.smithi119.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-24T11:31:37.424 INFO:teuthology.orchestra.run.smithi119.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-24T11:31:37.474 INFO:teuthology.orchestra.run.smithi119.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T11:31:37.541 INFO:teuthology.orchestra.run.smithi119.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-24T11:31:37.575 INFO:teuthology.orchestra.run.smithi119.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-24T11:31:37.608 INFO:teuthology.orchestra.run.smithi119.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-24T11:31:37.642 INFO:teuthology.orchestra.run.smithi119.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-24T11:31:37.709 INFO:teuthology.orchestra.run.smithi119.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-03-24T11:31:37.893 INFO:teuthology.orchestra.run.smithi119.stdout:(16/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86 20 MB/s | 17 MB 00:00 2024-03-24T11:31:37.910 INFO:teuthology.orchestra.run.smithi119.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-24T11:31:38.002 INFO:teuthology.orchestra.run.smithi119.stdout:(18/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-03-24T11:31:38.027 INFO:teuthology.orchestra.run.smithi119.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 371 kB/s | 43 kB 00:00 2024-03-24T11:31:38.098 INFO:teuthology.orchestra.run.smithi119.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 738 kB/s | 151 kB 00:00 2024-03-24T11:31:38.144 INFO:teuthology.orchestra.run.smithi119.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 232 kB/s | 11 kB 00:00 2024-03-24T11:31:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 977 kB/s | 172 kB 00:00 2024-03-24T11:31:38.203 INFO:teuthology.orchestra.run.smithi119.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-03-24T11:31:38.228 INFO:teuthology.orchestra.run.smithi119.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 464 kB/s | 23 kB 00:00 2024-03-24T11:31:38.254 INFO:teuthology.orchestra.run.smithi119.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-24T11:31:38.279 INFO:teuthology.orchestra.run.smithi119.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-24T11:31:38.304 INFO:teuthology.orchestra.run.smithi119.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-24T11:31:38.354 INFO:teuthology.orchestra.run.smithi119.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 617 kB/s | 46 kB 00:00 2024-03-24T11:31:38.396 INFO:teuthology.orchestra.run.smithi119.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 674 kB/s | 79 kB 00:00 2024-03-24T11:31:38.421 INFO:teuthology.orchestra.run.smithi119.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-24T11:31:38.472 INFO:teuthology.orchestra.run.smithi119.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-24T11:31:38.497 INFO:teuthology.orchestra.run.smithi119.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-24T11:31:38.522 INFO:teuthology.orchestra.run.smithi119.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-24T11:31:38.572 INFO:teuthology.orchestra.run.smithi119.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 402 kB/s | 20 kB 00:00 2024-03-24T11:31:38.598 INFO:teuthology.orchestra.run.smithi119.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-24T11:31:38.806 INFO:teuthology.orchestra.run.smithi119.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2024-03-24T11:31:38.810 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:38.810 INFO:teuthology.orchestra.run.smithi119.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-24T11:31:38.838 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:31:38.880 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:31:38.880 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:31:39.280 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:31:39.280 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:31:40.113 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:31:40.188 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:31:40.618 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:31:40.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T11:31:40.716 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T11:31:40.744 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T11:31:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T11:31:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T11:31:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T11:31:40.983 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T11:31:41.070 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T11:31:41.112 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T11:31:41.183 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T11:31:41.245 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T11:31:41.284 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T11:31:41.332 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T11:31:41.399 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T11:31:41.580 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T11:31:41.622 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T11:31:41.674 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T11:31:41.712 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T11:31:41.733 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T11:31:41.802 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T11:31:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:31:41.869 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T11:31:41.909 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T11:31:41.961 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T11:31:42.004 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T11:31:42.091 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T11:31:42.125 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T11:31:42.198 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T11:31:42.341 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T11:31:42.467 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:31:42.626 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T11:31:42.656 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:31:42.681 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:31:42.681 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:42.681 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:31:42.681 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:31:42.681 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:31:42.681 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:42.737 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:31:42.737 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T11:31:42.738 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T11:31:42.738 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:31:42.738 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:31:42.738 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:31:43.165 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:31:43.165 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:43.165 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:31:43.165 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:43.165 INFO:teuthology.orchestra.run.smithi114.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:31:43.166 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:43.166 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:31:43.166 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:31:43.166 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:31:43.166 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:43.166 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:31:43.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-03-24T11:31:43.993 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:41 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:31:44.098 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:31:44.124 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:31:44.124 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:44.124 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:31:44.124 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:31:44.125 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:31:44.125 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:44.163 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:31:44.165 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-24T11:31:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:31:44.165 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-24T11:31:44.165 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:31:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:31:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout:Install 36 Packages 2024-03-24T11:31:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:44.169 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 30 M 2024-03-24T11:31:44.169 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 107 M 2024-03-24T11:31:44.169 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:31:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:31:44.596 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:31:44.596 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:44.597 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:31:44.597 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:31:44.597 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:31:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:44.687 INFO:teuthology.orchestra.run.smithi114.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-24T11:31:44.816 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:31:44.839 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:31:44.839 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:44.839 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:31:44.839 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:31:44.839 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:31:44.839 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:44.845 INFO:teuthology.orchestra.run.smithi114.stdout:(2/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-03-24T11:31:44.887 INFO:teuthology.orchestra.run.smithi114.stdout:(3/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-03-24T11:31:44.896 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:31:44.996 INFO:teuthology.orchestra.run.smithi114.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 2.2 MB/s | 247 kB 00:00 2024-03-24T11:31:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-24T11:31:45.163 INFO:teuthology.orchestra.run.smithi114.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T11:31:45.188 INFO:teuthology.orchestra.run.smithi114.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-24T11:31:45.213 INFO:teuthology.orchestra.run.smithi114.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-24T11:31:45.239 INFO:teuthology.orchestra.run.smithi114.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-24T11:31:45.281 INFO:teuthology.orchestra.run.smithi114.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-24T11:31:45.356 INFO:teuthology.orchestra.run.smithi114.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-24T11:31:45.390 INFO:teuthology.orchestra.run.smithi114.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-24T11:31:45.423 INFO:teuthology.orchestra.run.smithi114.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-24T11:31:45.466 INFO:teuthology.orchestra.run.smithi114.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-24T11:31:45.741 INFO:teuthology.orchestra.run.smithi114.stdout:(15/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86 19 MB/s | 17 MB 00:00 2024-03-24T11:31:45.791 INFO:teuthology.orchestra.run.smithi114.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 384 kB/s | 125 kB 00:00 2024-03-24T11:31:45.859 INFO:teuthology.orchestra.run.smithi114.stdout:(17/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-03-24T11:31:46.075 INFO:teuthology.orchestra.run.smithi114.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 226 kB/s | 43 kB 00:00 2024-03-24T11:31:46.100 INFO:teuthology.orchestra.run.smithi114.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 489 kB/s | 151 kB 00:00 2024-03-24T11:31:46.159 INFO:teuthology.orchestra.run.smithi114.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 53 kB/s | 22 kB 00:00 2024-03-24T11:31:46.184 INFO:teuthology.orchestra.run.smithi114.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-24T11:31:46.209 INFO:teuthology.orchestra.run.smithi114.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-03-24T11:31:46.235 INFO:teuthology.orchestra.run.smithi114.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-24T11:31:46.260 INFO:teuthology.orchestra.run.smithi114.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-24T11:31:46.285 INFO:teuthology.orchestra.run.smithi114.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-03-24T11:31:46.311 INFO:teuthology.orchestra.run.smithi114.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-24T11:31:46.344 INFO:teuthology.orchestra.run.smithi114.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-03-24T11:31:46.370 INFO:teuthology.orchestra.run.smithi114.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 552 kB/s | 46 kB 00:00 2024-03-24T11:31:46.395 INFO:teuthology.orchestra.run.smithi114.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-03-24T11:31:46.429 INFO:teuthology.orchestra.run.smithi114.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-24T11:31:46.445 INFO:teuthology.orchestra.run.smithi114.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-24T11:31:46.471 INFO:teuthology.orchestra.run.smithi114.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-24T11:31:46.488 INFO:teuthology.orchestra.run.smithi114.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2024-03-24T11:31:46.563 INFO:teuthology.orchestra.run.smithi114.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T11:31:46.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T11:31:46.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T11:31:46.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T11:31:47.271 INFO:teuthology.orchestra.run.smithi114.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 25 kB/s | 20 kB 00:00 2024-03-24T11:31:47.297 INFO:teuthology.orchestra.run.smithi114.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 517 kB/s | 427 kB 00:00 2024-03-24T11:31:47.302 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:47.302 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.5 MB/s | 30 MB 00:03 2024-03-24T11:31:47.326 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:31:47.368 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:31:47.368 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:31:47.658 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:31:47.659 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:31:47.661 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:47.661 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:47.779 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:31:47.780 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:31:48.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-03-24T11:31:48.564 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:52 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:31:48.659 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:31:48.710 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:31:48.711 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:31:48.711 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:48.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-03-24T11:31:49.103 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:31:49.158 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T11:31:49.190 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T11:31:49.218 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T11:31:49.291 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T11:31:49.336 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T11:31:49.344 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:53 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:31:49.369 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T11:31:49.464 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T11:31:49.503 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:31:49.504 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:31:49.504 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:49.504 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 748 k 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 754 k 2024-03-24T11:31:49.505 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:31:49.549 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T11:31:49.598 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T11:31:49.687 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T11:31:49.750 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T11:31:49.794 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T11:31:49.853 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T11:31:49.922 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T11:31:49.965 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 1.6 MB/s | 748 kB 00:00 2024-03-24T11:31:49.965 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:49.966 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.6 MB/s | 748 kB 00:00 2024-03-24T11:31:49.966 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:31:49.971 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:31:49.971 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:31:49.976 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:31:49.977 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:31:50.048 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:31:50.115 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T11:31:50.157 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T11:31:50.203 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T11:31:50.247 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T11:31:50.285 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T11:31:50.365 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T11:31:50.395 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:31:50.429 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T11:31:50.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T11:31:50.513 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T11:31:50.514 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:50.547 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T11:31:50.587 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T11:31:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T11:31:50.760 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T11:31:50.909 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T11:31:50.970 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:51.034 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:31:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T11:31:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:51.198 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:31:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:31:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:51.198 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:31:51.249 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:31:51.249 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:51.249 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:31:51.249 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:31:51.249 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:31:51.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:51.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:31:51.974 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:55 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:31:52.122 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:31:52.123 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:52.123 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-24T11:31:52.123 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:52.123 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:31:52.123 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T11:31:52.123 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:52.124 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:31:52.124 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:31:52.124 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:31:52.124 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:52.124 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 145 k 2024-03-24T11:31:52.124 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 436 k 2024-03-24T11:31:52.124 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:31:52.462 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 430 kB/s | 145 kB 00:00 2024-03-24T11:31:52.463 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:52.463 INFO:teuthology.orchestra.run.smithi119.stdout:Total 428 kB/s | 145 kB 00:00 2024-03-24T11:31:52.464 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:31:52.471 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:31:52.471 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:31:52.524 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:31:52.524 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:31:52.659 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:31:52.692 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:31:52.713 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:31:52.717 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:31:52.717 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:52.718 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:31:52.718 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:31:52.718 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:31:52.718 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:52.737 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:31:52.737 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:52.738 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:31:52.738 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:53.168 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:31:53.191 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:31:53.191 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:53.191 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:31:53.191 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:31:53.191 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:31:53.191 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:53.410 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:31:53.433 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:31:53.433 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:31:53.433 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:31:53.433 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:31:53.433 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:31:53.433 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:53.508 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:31:53.731 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:31:53.731 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:53.731 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:31:53.731 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:53.732 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:53.732 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:54.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-03-24T11:31:54.604 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:58 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:31:54.702 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:31:54.753 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:31:54.754 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:31:54.754 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:54.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:31:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:31:55.232 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T11:31:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T11:31:55.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T11:31:55.235 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T11:31:55.236 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T11:31:55.376 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:59 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:31:55.527 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:31:55.528 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T11:31:55.529 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:31:55.530 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:31:55.530 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:55.530 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:31:55.530 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-03-24T11:31:55.530 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-03-24T11:31:55.530 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:55.530 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 6.9 M 2024-03-24T11:31:55.531 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-03-24T11:31:55.531 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:31:56.074 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 82 kB/s | 15 kB 00:00 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:31:56.078 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:31:56.079 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:56.080 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:31:56.133 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1667.gdb 128 kB/s | 31 kB 00:00 2024-03-24T11:31:56.224 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-24T11:31:56.250 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-24T11:31:56.358 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.3 MB/s | 863 kB 00:00 2024-03-24T11:31:56.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-03-24T11:31:56.442 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 3.2 MB/s | 1.8 MB 00:00 2024-03-24T11:31:56.501 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-24T11:31:56.559 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-24T11:31:56.610 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 616 kB/s | 31 kB 00:00 2024-03-24T11:31:56.668 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-24T11:31:56.752 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 576 kB/s | 144 kB 00:00 2024-03-24T11:31:56.793 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.8 MB/s | 2.0 MB 00:00 2024-03-24T11:31:56.796 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:56.796 INFO:teuthology.orchestra.run.smithi119.stdout:Total 5.4 MB/s | 6.9 MB 00:01 2024-03-24T11:31:56.814 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:31:56.830 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:31:56.831 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:31:56.976 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:31:56.999 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:31:56.999 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:31:57.072 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:31:57.123 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:31:57.125 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:31:57.125 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:31:57.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-03-24T11:31:57.510 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:31:57.585 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T11:31:57.654 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T11:31:57.700 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T11:31:57.731 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T11:31:57.740 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:55 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:31:57.824 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T11:31:57.888 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:31:57.901 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:31:57.902 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:57.902 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 748 k 2024-03-24T11:31:57.902 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 754 k 2024-03-24T11:31:57.902 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:31:57.946 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T11:31:58.040 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T11:31:58.084 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T11:31:58.122 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:31:58.154 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T11:31:58.325 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-24T11:31:58.326 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:31:58.326 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-24T11:31:58.326 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:31:58.331 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:31:58.332 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:31:58.337 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:31:58.337 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:31:58.394 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T11:31:58.410 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:31:58.421 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:31:58.808 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:58.889 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:59.219 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T11:31:59.220 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T11:31:59.266 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:59.537 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:31:59.538 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:59.538 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:31:59.538 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:31:59.538 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:31:59.538 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:31:59.563 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T11:31:59.564 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:31:59.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:31:59.565 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:31:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:31:59.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:31:59.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:32:00.304 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:58 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:00.382 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:04 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:00.458 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:00.458 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 145 k 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 436 k 2024-03-24T11:32:00.459 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:00.535 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:32:00.536 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-03-24T11:32:00.537 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:00.538 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 37 M 2024-03-24T11:32:00.538 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 209 M 2024-03-24T11:32:00.538 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:00.665 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 708 kB/s | 145 kB 00:00 2024-03-24T11:32:00.665 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:00.666 INFO:teuthology.orchestra.run.smithi114.stdout:Total 702 kB/s | 145 kB 00:00 2024-03-24T11:32:00.666 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:00.674 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:00.674 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:00.849 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2024-03-24T11:32:00.865 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:00.883 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 524 kB/s | 17 kB 00:00 2024-03-24T11:32:00.925 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 846 kB/s | 34 kB 00:00 2024-03-24T11:32:00.942 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:32:00.965 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:32:00.965 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:32:00.965 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:32:00.965 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:00.983 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T11:32:01.150 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-24T11:32:01.217 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-24T11:32:01.242 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-24T11:32:01.392 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-16 11 MB/s | 7.4 MB 00:00 2024-03-24T11:32:01.485 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-03-24T11:32:01.643 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-24T11:32:01.668 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 712 kB/s | 195 kB 00:00 2024-03-24T11:32:01.932 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:32:01.932 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:01.932 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:01.932 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:01.932 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:01.932 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:02.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-03-24T11:32:02.243 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 324 kB/s | 245 kB 00:00 2024-03-24T11:32:02.247 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:02.247 INFO:teuthology.orchestra.run.smithi119.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-24T11:32:02.291 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:02.317 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:02.317 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:02.671 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:02.671 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:02.783 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:00 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:02.880 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:02.933 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:02.934 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:02.934 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:03.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:32:03.376 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:03.458 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:32:03.498 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:32:03.564 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:01 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:03.590 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:32:03.626 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:32:03.714 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:03.715 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-24T11:32:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:32:03.715 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-24T11:32:03.715 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:32:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:03.717 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:03.717 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-24T11:32:03.717 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-03-24T11:32:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:03.717 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 6.9 M 2024-03-24T11:32:03.717 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-03-24T11:32:03.717 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:03.803 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:32:03.833 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:32:03.920 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1667.gdb 165 kB/s | 31 kB 00:00 2024-03-24T11:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 54 kB/s | 15 kB 00:00 2024-03-24T11:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-24T11:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-24T11:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 4.4 MB/s | 1.8 MB 00:00 2024-03-24T11:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-03-24T11:32:04.567 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:32:04.679 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-03-24T11:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-24T11:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 867 kB/s | 31 kB 00:00 2024-03-24T11:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.9 MB/s | 188 kB 00:00 2024-03-24T11:32:04.996 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 541 kB/s | 144 kB 00:00 2024-03-24T11:32:05.338 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.5 MB/s | 2.0 MB 00:00 2024-03-24T11:32:05.343 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:05.343 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.2 MB/s | 6.9 MB 00:01 2024-03-24T11:32:05.361 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:05.377 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:05.377 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:05.541 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:05.541 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:32:06.048 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T11:32:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T11:32:06.211 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T11:32:06.242 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T11:32:06.345 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T11:32:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T11:32:06.569 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T11:32:06.613 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T11:32:06.642 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:32:06.674 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T11:32:06.918 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T11:32:06.950 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:32:06.958 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:32:06.974 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:32:07.742 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T11:32:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:08.088 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:08.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:32:08.894 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:06 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:09.044 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:09.045 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-24T11:32:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:32:09.045 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-24T11:32:09.045 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:09.046 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:09.047 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-24T11:32:09.047 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-03-24T11:32:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:09.047 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 37 M 2024-03-24T11:32:09.047 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 209 M 2024-03-24T11:32:09.047 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:09.387 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 228 kB/s | 32 kB 00:00 2024-03-24T11:32:09.421 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 531 kB/s | 17 kB 00:00 2024-03-24T11:32:09.454 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-24T11:32:09.563 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.5 MB/s | 3.0 MB 00:00 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:32:09.585 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:32:09.586 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:32:09.586 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:32:09.586 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:32:09.638 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-24T11:32:09.721 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 4.8 MB 00:00 2024-03-24T11:32:09.747 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-24T11:32:10.047 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:32:10.059 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:10.060 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:10.264 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-16 7.3 MB/s | 7.4 MB 00:01 2024-03-24T11:32:10.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:32:10.548 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 391 kB/s | 195 kB 00:00 2024-03-24T11:32:10.589 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 956 kB/s | 803 kB 00:00 2024-03-24T11:32:11.027 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:15 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:11.179 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:32:11.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-03-24T11:32:11.181 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:11.182 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.0 M 2024-03-24T11:32:11.182 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-03-24T11:32:11.182 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:11.190 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 265 kB/s | 245 kB 00:00 2024-03-24T11:32:11.193 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:11.193 INFO:teuthology.orchestra.run.smithi114.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-24T11:32:11.234 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:11.260 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:11.260 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:11.358 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2024-03-24T11:32:11.383 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-03-24T11:32:11.434 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-24T11:32:11.459 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T11:32:11.484 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el9 192 kB/s | 49 kB 00:00 2024-03-24T11:32:11.510 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-24T11:32:11.535 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 568 kB/s | 43 kB 00:00 2024-03-24T11:32:11.585 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 200 kB/s | 15 kB 00:00 2024-03-24T11:32:11.612 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 315 kB/s | 32 kB 00:00 2024-03-24T11:32:11.617 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:11.618 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:11.663 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-24T11:32:11.688 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-24T11:32:11.713 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-24T11:32:11.755 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.0 MB/s | 1.0 MB 00:00 2024-03-24T11:32:11.757 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:11.757 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.6 MB/s | 2.0 MB 00:00 2024-03-24T11:32:11.757 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:11.771 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:11.771 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:11.862 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:11.864 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:12.316 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:12.387 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:32:12.419 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:32:12.516 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:32:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:32:12.556 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:32:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:32:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:32:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:32:12.732 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:32:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:32:12.771 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:32:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:32:12.848 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:32:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:32:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:32:13.514 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:32:13.636 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:32:13.719 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:32:13.763 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:32:13.797 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:32:13.845 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:32:13.867 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:32:14.638 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:32:14.639 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:32:14.639 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:32:14.911 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:32:14.996 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:32:14.996 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:14.996 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:14.996 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:14.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:32:14.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:32:14.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:14.997 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:15.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:32:15.796 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:19 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:15.946 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:15.946 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-24T11:32:15.946 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-03-24T11:32:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:15.948 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:32:15.948 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 6.8 M 2024-03-24T11:32:15.948 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 33 M 2024-03-24T11:32:15.949 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:15.970 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:32:16.423 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T11:32:16.448 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 867 kB/s | 248 kB 00:00 2024-03-24T11:32:16.540 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 365 kB/s | 138 kB 00:00 2024-03-24T11:32:16.824 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-03-24T11:32:16.882 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-24T11:32:17.066 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 853 kB/s | 548 kB 00:00 2024-03-24T11:32:17.067 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:17.067 INFO:teuthology.orchestra.run.smithi119.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-03-24T11:32:17.081 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:17.091 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:17.091 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:17.179 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:17.179 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:17.476 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:17.654 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:32:17.717 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:32:18.239 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:32:18.320 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:32:18.381 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:32:18.404 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:32:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:32:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:32:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:32:18.966 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:19.166 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:32:19.166 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T11:32:19.167 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:32:19.167 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:32:19.167 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:32:19.167 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:32:19.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:32:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:19.454 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:19.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-03-24T11:32:19.924 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:17 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:20.075 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:32:20.076 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-03-24T11:32:20.077 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:20.078 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.0 M 2024-03-24T11:32:20.078 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-03-24T11:32:20.078 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:20.271 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:24 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:20.338 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 213 kB/s | 24 kB 00:00 2024-03-24T11:32:20.363 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-03-24T11:32:20.405 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-24T11:32:20.420 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:20.420 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:20.420 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:32:20.420 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-03-24T11:32:20.421 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:20.422 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 964 k 2024-03-24T11:32:20.422 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 2.8 M 2024-03-24T11:32:20.422 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:20.430 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-24T11:32:20.455 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-24T11:32:20.472 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-24T11:32:20.523 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 299 kB/s | 15 kB 00:00 2024-03-24T11:32:20.540 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 385 kB/s | 32 kB 00:00 2024-03-24T11:32:20.541 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 657 kB/s | 79 kB 00:00 2024-03-24T11:32:20.557 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el9 148 kB/s | 49 kB 00:00 2024-03-24T11:32:20.598 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 2.6 MB/s | 203 kB 00:00 2024-03-24T11:32:20.632 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-03-24T11:32:20.649 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-03-24T11:32:20.682 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-03-24T11:32:20.684 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:20.684 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2024-03-24T11:32:20.684 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:20.697 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:20.697 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:20.733 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 2.8 MB/s | 886 kB 00:00 2024-03-24T11:32:20.734 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:20.735 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-03-24T11:32:20.735 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:20.756 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:20.756 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:20.785 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:20.787 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:20.831 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:20.832 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:21.018 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:21.160 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:32:21.233 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:32:21.291 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:21.463 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:32:21.517 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:32:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:32:21.627 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:32:21.668 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:32:21.761 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:32:21.862 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:32:21.911 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:32:21.942 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:32:21.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T11:32:22.261 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:32:22.261 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:22.261 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:22.261 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:32:22.262 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:22.262 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:22.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-volume 2024-03-24T11:32:22.643 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:32:22.691 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:32:22.725 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:32:22.773 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:32:22.795 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:32:23.150 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:27 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:23.304 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:23.304 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-03-24T11:32:23.305 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:23.306 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 332 k 2024-03-24T11:32:23.306 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 1.5 M 2024-03-24T11:32:23.306 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:23.429 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 559 kB/s | 69 kB 00:00 2024-03-24T11:32:23.530 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:32:23.530 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T11:32:23.530 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:32:23.532 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:32:23.533 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:32:23.533 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:32:23.579 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 966 kB/s | 264 kB 00:00 2024-03-24T11:32:23.580 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:23.581 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-24T11:32:23.581 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:23.589 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:23.589 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:23.629 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:23.629 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:23.775 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:23.876 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:32:23.877 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:32:23.878 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:32:23.878 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:23.878 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:23.956 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T11:32:23.999 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:32:24.014 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:32:24.014 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:32:24.014 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T11:32:24.014 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:24.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:32:24.671 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:22 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:24.746 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T11:32:24.821 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:24.822 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-24T11:32:24.822 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-24T11:32:24.823 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-03-24T11:32:24.824 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:24.824 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 6.8 M 2024-03-24T11:32:24.824 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-03-24T11:32:24.824 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:25.060 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T11:32:25.060 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:25.060 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:25.060 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:25.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T11:32:25.060 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:25.060 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:25.285 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T11:32:25.326 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 449 kB/s | 138 kB 00:00 2024-03-24T11:32:25.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-03-24T11:32:25.368 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 710 kB/s | 248 kB 00:00 2024-03-24T11:32:25.794 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 124 kB/s | 58 kB 00:00 2024-03-24T11:32:25.844 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 113 kB/s | 51 kB 00:00 2024-03-24T11:32:25.932 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:29 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:26.044 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 722 kB/s | 548 kB 00:00 2024-03-24T11:32:26.048 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:26.049 INFO:teuthology.orchestra.run.smithi114.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-24T11:32:26.061 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:26.072 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:26.072 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:26.083 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:26.084 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:26.084 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-24T11:32:26.084 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:26.084 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:26.084 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T11:32:26.084 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:26.084 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:26.085 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:26.085 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:32:26.085 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:26.085 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 127 k 2024-03-24T11:32:26.085 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 456 k 2024-03-24T11:32:26.085 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:26.161 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:26.162 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:26.402 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 400 kB/s | 127 kB 00:00 2024-03-24T11:32:26.403 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:26.403 INFO:teuthology.orchestra.run.smithi119.stdout:Total 398 kB/s | 127 kB 00:00 2024-03-24T11:32:26.403 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:26.408 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:26.408 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:26.458 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:26.465 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:26.465 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:26.629 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:32:26.654 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:26.688 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:32:26.759 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:27.233 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:32:27.324 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:32:27.394 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:32:27.425 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:32:27.459 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:27.459 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:27.459 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:27.459 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:27.459 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:27.459 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:27.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-03-24T11:32:28.255 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:28.291 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:32:28.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T11:32:28.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:32:28.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:32:28.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:32:28.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:32:28.355 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:28.408 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:28.409 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:32:28.410 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:28.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-03-24T11:32:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:32:28.629 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:28.630 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:28.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-03-24T11:32:29.020 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:33 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:29.170 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:29.171 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 31 k 2024-03-24T11:32:29.172 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 140 k 2024-03-24T11:32:29.172 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:29.330 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 197 kB/s | 31 kB 00:00 2024-03-24T11:32:29.330 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:29.330 INFO:teuthology.orchestra.run.smithi119.stdout:Total 196 kB/s | 31 kB 00:00 2024-03-24T11:32:29.330 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:29.334 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:29.334 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:29.362 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:29.363 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:29.422 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:27 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:29.458 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:29.569 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:29.571 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:29.572 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:29.573 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:29.573 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-03-24T11:32:29.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:29.573 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 964 k 2024-03-24T11:32:29.573 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.8 M 2024-03-24T11:32:29.573 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:29.696 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 641 kB/s | 79 kB 00:00 2024-03-24T11:32:29.947 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:30.121 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 1.6 MB/s | 886 kB 00:00 2024-03-24T11:32:30.123 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:30.123 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.7 MB/s | 964 kB 00:00 2024-03-24T11:32:30.123 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:30.144 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:30.144 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:30.202 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:30.202 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:30.202 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:30.202 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:30.202 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:30.202 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:30.216 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:30.217 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:30.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-03-24T11:32:30.397 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:30.540 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:32:30.612 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:32:30.919 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:31.016 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:31.067 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:31.068 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:32:31.068 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:31.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-03-24T11:32:31.297 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:32:31.297 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T11:32:31.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:32:31.578 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:31.578 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:31.578 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:32:31.578 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:31.578 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:31.679 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:31.775 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:31.826 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:31.827 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:32:31.828 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:31.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-03-24T11:32:31.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-volume 2024-03-24T11:32:32.453 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:36 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:32.497 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:30 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:32.549 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:32.600 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:32.601 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:32:32.601 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-24T11:32:32.655 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T11:32:32.656 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:32.656 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:32.656 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:32.656 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-03-24T11:32:32.656 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:32.656 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 332 k 2024-03-24T11:32:32.657 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 1.5 M 2024-03-24T11:32:32.657 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:32.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-03-24T11:32:32.784 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 540 kB/s | 69 kB 00:00 2024-03-24T11:32:32.934 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 951 kB/s | 264 kB 00:00 2024-03-24T11:32:32.935 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:32.936 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-24T11:32:32.936 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:32.944 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:32.944 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:32.979 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:32.979 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:33.222 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:33.318 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:33.318 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T11:32:33.354 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:32:33.369 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:32:33.369 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:32:33.369 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T11:32:33.369 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:33.371 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:33.372 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:32:33.372 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:33.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-03-24T11:32:33.990 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T11:32:34.086 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:34.139 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:34.140 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:32:34.140 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:34.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-03-24T11:32:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T11:32:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:34.400 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:32:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T11:32:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:34.400 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:34.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-03-24T11:32:34.771 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:34.867 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:34.918 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:34.919 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-24T11:32:34.919 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:34.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-03-24T11:32:35.301 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:33 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:35.451 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 127 k 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-03-24T11:32:35.452 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:35.532 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:35.678 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:35.679 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:32:35.680 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:35.680 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 85 k 2024-03-24T11:32:35.680 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 231 k 2024-03-24T11:32:35.680 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:35.923 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 269 kB/s | 127 kB 00:00 2024-03-24T11:32:35.923 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:35.924 INFO:teuthology.orchestra.run.smithi114.stdout:Total 268 kB/s | 127 kB 00:00 2024-03-24T11:32:35.924 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:35.929 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:35.929 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:35.985 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:35.985 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:36.165 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 175 kB/s | 85 kB 00:00 2024-03-24T11:32:36.166 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:36.166 INFO:teuthology.orchestra.run.smithi119.stdout:Total 175 kB/s | 85 kB 00:00 2024-03-24T11:32:36.166 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:36.173 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:36.173 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:36.191 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:36.230 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:36.230 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:36.290 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:36.369 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:36.480 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:36.772 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:36.842 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:37.073 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:37.074 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:37.074 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:37.074 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:37.074 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:37.074 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:37.116 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:37.116 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:37.116 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:37.116 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:37.116 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:37.116 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:37.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-03-24T11:32:37.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-03-24T11:32:37.863 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:37.961 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:37.965 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:38.015 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:38.016 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:38.017 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:38.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-03-24T11:32:38.121 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:32:38.122 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:38.123 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.1 M 2024-03-24T11:32:38.123 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 13 M 2024-03-24T11:32:38.123 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:38.625 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:36 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:38.772 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:38.773 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:38.773 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-24T11:32:38.773 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:38.773 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 31 k 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 140 k 2024-03-24T11:32:38.774 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:39.243 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 66 kB/s | 31 kB 00:00 2024-03-24T11:32:39.244 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:39.244 INFO:teuthology.orchestra.run.smithi114.stdout:Total 66 kB/s | 31 kB 00:00 2024-03-24T11:32:39.244 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:39.247 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:39.247 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:39.279 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:39.279 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:39.373 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:39.511 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 2.2 MB/s | 3.1 MB 00:01 2024-03-24T11:32:39.512 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:39.512 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.2 MB/s | 3.1 MB 00:01 2024-03-24T11:32:39.513 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:39.524 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:39.524 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:39.601 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:39.601 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:39.926 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:40.028 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:40.095 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:40.119 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:40.119 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:32:40.119 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:32:40.120 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:32:40.120 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:32:40.120 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:40.182 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:40.182 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:40.182 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:40.182 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:40.182 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:40.182 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:40.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-03-24T11:32:40.948 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:41.045 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:41.098 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:41.099 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:41.099 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:41.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-03-24T11:32:41.250 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:41.250 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:41.250 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:41.250 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:41.250 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:41.250 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:41.501 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-03-24T11:32:41.721 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:41.817 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:41.867 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:41.868 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:41.869 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:41.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-03-24T11:32:42.075 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:46 ago on Sun 24 Mar 2024 11:30:56 AM UTC. 2024-03-24T11:32:42.222 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-24T11:32:42.223 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:42.223 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-03-24T11:32:42.223 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 171 k 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 494 k 2024-03-24T11:32:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-24T11:32:42.487 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:42.582 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:42.614 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 438 kB/s | 171 kB 00:00 2024-03-24T11:32:42.615 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:42.615 INFO:teuthology.orchestra.run.smithi119.stdout:Total 436 kB/s | 171 kB 00:00 2024-03-24T11:32:42.615 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-24T11:32:42.622 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-24T11:32:42.622 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-24T11:32:42.633 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:42.634 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:42.634 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:42.679 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-24T11:32:42.679 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-24T11:32:42.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-03-24T11:32:42.826 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-24T11:32:42.937 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:43.259 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:43.355 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:43.397 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:43.406 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:43.407 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:43.407 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:43.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-03-24T11:32:43.669 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:43.670 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:43.670 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-24T11:32:43.670 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:43.670 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:32:43.670 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-24T11:32:43.907 DEBUG:teuthology.parallel:result is None 2024-03-24T11:32:44.036 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:42 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:44.133 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:44.185 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:44.186 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:44.186 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:44.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-03-24T11:32:44.800 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:42 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:44.896 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:32:44.948 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:44.949 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-24T11:32:44.950 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:45.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-03-24T11:32:45.567 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:43 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:32:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:45.717 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-03-24T11:32:45.717 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 231 k 2024-03-24T11:32:45.717 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:45.921 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 417 kB/s | 85 kB 00:00 2024-03-24T11:32:45.922 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:45.922 INFO:teuthology.orchestra.run.smithi114.stdout:Total 414 kB/s | 85 kB 00:00 2024-03-24T11:32:45.922 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:45.929 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:45.929 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:45.983 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:45.984 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:46.124 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:46.221 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:46.618 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:46.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:46.909 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:46.909 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:46.909 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:46.909 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:46.910 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:47.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-03-24T11:32:47.755 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:45 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:47.908 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:47.909 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:47.909 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-24T11:32:47.909 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:47.909 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:47.909 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.1 M 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 13 M 2024-03-24T11:32:47.910 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:48.710 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 3.9 MB/s | 3.1 MB 00:00 2024-03-24T11:32:48.710 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:48.711 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.9 MB/s | 3.1 MB 00:00 2024-03-24T11:32:48.711 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:48.720 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:48.721 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:48.794 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:48.794 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:49.214 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:49.286 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:49.311 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:49.311 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:32:49.311 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:32:49.311 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:32:49.311 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:32:49.311 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:50.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:50.388 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:50.389 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:50.389 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:50.389 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:50.389 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:50.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-03-24T11:32:51.233 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:49 ago on Sun 24 Mar 2024 11:31:02 AM UTC. 2024-03-24T11:32:51.381 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-24T11:32:51.382 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-24T11:32:51.382 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:51.382 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-24T11:32:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T11:32:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:51.383 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-24T11:32:51.383 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-24T11:32:51.383 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-24T11:32:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:51.383 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 171 k 2024-03-24T11:32:51.383 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 494 k 2024-03-24T11:32:51.384 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-24T11:32:51.738 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 481 kB/s | 171 kB 00:00 2024-03-24T11:32:51.739 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:32:51.740 INFO:teuthology.orchestra.run.smithi114.stdout:Total 479 kB/s | 171 kB 00:00 2024-03-24T11:32:51.740 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-24T11:32:51.748 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-24T11:32:51.748 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-24T11:32:51.803 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-24T11:32:51.804 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-24T11:32:51.949 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-24T11:32:52.053 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:52.480 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:52.764 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:32:52.764 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:52.764 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-24T11:32:52.764 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:32:52.764 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:32:52.764 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-24T11:32:53.017 DEBUG:teuthology.parallel:result is None 2024-03-24T11:32:53.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:32:53.158 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:32:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T11:32:53.179 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T11:32:53.179 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T11:32:53.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:32:53.318 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:32:53.342 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T11:32:53.342 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T11:32:53.343 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T11:32:53.344 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T11:32:53.344 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:32:53.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:32:53.372 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:32:53.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:32:53.412 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T11:32:53.412 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:32:53.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:32:53.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:32:53.506 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:32:53.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:32:53.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:32:53.604 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T11:32:53.604 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:32:53.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:32:53.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:32:53.698 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:32:53.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:32:53.728 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:32:53.796 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T11:32:53.796 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:32:53.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:32:53.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:32:53.889 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:32:53.890 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:32:53.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:32:53.985 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:32:54.090 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'mon_bind_msgr2': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'cephadm_mode': 'cephadm-package'} 2024-03-24T11:32:54.090 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:32:54.091 INFO:tasks.cephadm:Cluster fsid is 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:32:54.091 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:32:54.091 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.114:6789]', 'mon.c': '[v1:172.21.15.114:6790]', 'mon.b': '[v1:172.21.15.119:6789]'} 2024-03-24T11:32:54.091 INFO:tasks.cephadm:First mon is mon.a on smithi114 2024-03-24T11:32:54.091 INFO:tasks.cephadm:First mgr is y 2024-03-24T11:32:54.091 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:32:54.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-03-24T11:32:54.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-03-24T11:32:54.157 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on all hosts... 2024-03-24T11:32:54.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:32:54.162 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:32:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:32:55.048 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:33:30.658 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-24T11:33:30.658 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:33:30.658 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:33:30.658 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-03-24T11:33:30.658 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:33:30.658 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-24T11:33:30.658 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-24T11:33:32.827 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-24T11:33:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:33:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:33:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-03-24T11:33:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:33:32.827 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-24T11:33:32.827 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-24T11:33:32.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-03-24T11:33:32.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-03-24T11:33:32.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-03-24T11:33:32.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-03-24T11:33:32.986 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:33:32.988 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:33:32.988 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:33:32.988 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T11:33:32.988 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-24T11:33:32.989 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-24T11:33:32.989 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-24T11:33:32.989 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:33:32.989 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:33:32.989 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:33:32.989 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-24T11:33:32.990 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:33:32.990 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:33:32.990 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-24T11:33:32.990 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-24T11:33:32.990 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T11:33:32.991 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:33:32.991 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:33:33.016 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 412c996a-e9d2-11ee-95cd-87774f69a715 mon election default strategy = 1 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-24T11:33:33.016 DEBUG:teuthology.orchestra.run.smithi114:mon.a> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T11:33:33.059 DEBUG:teuthology.orchestra.run.smithi114:mgr.y> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T11:33:33.102 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:33:33.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 -v bootstrap --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.114:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:33:33.295 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:33:33.295 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69', '-v', 'bootstrap', '--fsid', '412c996a-e9d2-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.114:6789]', '--skip-admin-label'] 2024-03-24T11:33:33.321 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:33:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T11:33:33.321 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-03-24T11:33:33.345 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:33:33.345 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-03-24T11:33:33.345 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-03-24T11:33:33.352 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:33:33.352 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:33:33.358 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:33:33.358 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-03-24T11:33:33.364 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-03-24T11:33:33.370 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-03-24T11:33:33.370 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:33:33.370 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-03-24T11:33:33.394 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:33:33.394 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:33:33.394 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-03-24T11:33:33.394 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-03-24T11:33:33.400 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:33:33.400 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:33:33.405 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:33:33.406 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-03-24T11:33:33.411 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-03-24T11:33:33.418 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-03-24T11:33:33.418 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:33:33.418 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-03-24T11:33:33.418 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:33.418 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 140546955802224 on /run/cephadm/412c996a-e9d2-11ee-95cd-87774f69a715.lock 2024-03-24T11:33:33.418 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 140546955802224 acquired on /run/cephadm/412c996a-e9d2-11ee-95cd-87774f69a715.lock 2024-03-24T11:33:33.418 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-03-24T11:33:33.419 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:6789], mon addrv is [v1:172.21.15.114:6789] 2024-03-24T11:33:33.422 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-03-24T11:33:33.422 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-03-24T11:33:33.424 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:33:33.424 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T11:33:33.424 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2024-03-24T11:33:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:33:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T11:33:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:33:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T11:33:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-03-24T11:33:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:33:33.427 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-03-24T11:33:33.428 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-24T11:33:33.428 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:33:33.429 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:33:33.894 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:33:33.894 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:33:33.894 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T11:33:33.894 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:989de1c398e87dbcb4906791a352373dc0e0415d619bc5b36f1f1823df929cb1 2024-03-24T11:33:33.894 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-24T11:33:33.894 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:33:33.894 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T11:33:39.742 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:33:39.742 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:33:39.742 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T11:33:45.966 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-03-24T11:33:45.966 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-03-24T11:33:47.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCaDwBmbcT9ERAAKAprybm2kGRHX3ETGG6IjA== 2024-03-24T11:33:47.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCbDwBmI5xOFhAAEbvVdujanvQqceSFpxGL5A== 2024-03-24T11:33:48.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCcDwBm65AmEhAAbNybWlpZHahfvbmJw5Rt1Q== 2024-03-24T11:33:48.826 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-03-24T11:33:49.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:33:49.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T11:33:49.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:49.749 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:33:49.749 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for a [v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:33:49.749 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-03-24T11:33:49.749 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:49.749 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:33:49.749 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:33:49.749 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-03-24T11:33:50.554 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.a on 2024-03-24T11:33:50.757 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T11:33:51.029 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:33:51.600 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-412c996a-e9d2-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-412c996a-e9d2-11ee-95cd-87774f69a715.target. 2024-03-24T11:33:51.600 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-412c996a-e9d2-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-412c996a-e9d2-11ee-95cd-87774f69a715.target. 2024-03-24T11:33:52.286 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:33:52.286 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T11:33:52.506 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-412c996a-e9d2-11ee-95cd-87774f69a715.target.wants/ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-412c996a-e9d2-11ee-95cd-87774f69a715@.service. 2024-03-24T11:33:52.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:52 smithi114 systemd[1]: Starting Ceph mon.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:33:52.844 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:52 smithi114 podman[29787]: 2024-03-24 11:33:52.727683224 +0000 UTC m=+0.099944465 container create b0dd44c094815a10aead92c0fbba78277232460454e417c5df1107629fb1d14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T11:33:52.844 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:52 smithi114 podman[29787]: 2024-03-24 11:33:52.645310821 +0000 UTC m=+0.017572074 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:33:53.060 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:33:53.060 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-03-24T11:33:53.066 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:33:53.066 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-03-24T11:33:53.067 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-03-24T11:33:53.067 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:33:53.067 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-03-24T11:33:53.067 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-03-24T11:33:53.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:52 smithi114 podman[29787]: 2024-03-24 11:33:52.930202152 +0000 UTC m=+0.302463412 container init b0dd44c094815a10aead92c0fbba78277232460454e417c5df1107629fb1d14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T11:33:53.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:52 smithi114 podman[29787]: 2024-03-24 11:33:52.935666246 +0000 UTC m=+0.307927487 container start b0dd44c094815a10aead92c0fbba78277232460454e417c5df1107629fb1d14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:33:53.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:53 smithi114 ceph-mon[29825]: mkfs 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:53.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:53 smithi114 ceph-mon[29825]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:33:53.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:53 smithi114 bash[29787]: b0dd44c094815a10aead92c0fbba78277232460454e417c5df1107629fb1d14c 2024-03-24T11:33:53.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:53 smithi114 systemd[1]: Started Ceph mon.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:33:54.103 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:54 smithi114 ceph-mon[29825]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:33:54.103 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:54 smithi114 ceph-mon[29825]: monmap e1: 1 mons at {a=v1:172.21.15.114:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:33:54.103 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:54 smithi114 ceph-mon[29825]: fsmap 2024-03-24T11:33:54.103 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:54 smithi114 ceph-mon[29825]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:33:54.104 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:54 smithi114 ceph-mon[29825]: mgrmap e1: no daemons active 2024-03-24T11:33:54.104 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:54 smithi114 ceph-mon[29825]: from='client.? v1:172.21.15.114:0/2910942233' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T11:33:54.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T11:33:54.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:54.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T11:33:54.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.607855s) 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.525+0000 7f7ebc862700 1 Processor -- start 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.525+0000 7f7ebc862700 1 -- start start 2024-03-24T11:33:54.112 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.525+0000 7f7ebc862700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7eb8079720 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.525+0000 7f7eb659c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7eb81043b0 0x7f7eb81047d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39104/0 (socket says 172.21.15.114:39104) 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.525+0000 7f7eb659c700 1 -- 172.21.15.114:0/1537257831 learned_addr learned my addr 172.21.15.114:0/1537257831 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7eb559a700 1 -- 172.21.15.114:0/1537257831 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079067759 0 0) 0x7f7eb8079720 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7eb559a700 1 -- 172.21.15.114:0/1537257831 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea00035b0 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7eb559a700 1 -- 172.21.15.114:0/1537257831 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2168412671 0 0) 0x7f7ea00035b0 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7eb559a700 1 -- 172.21.15.114:0/1537257831 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb8105ef0 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7eb559a700 1 -- 172.21.15.114:0/1537257831 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f7ea8001580 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7eb559a700 1 -- 172.21.15.114:0/1537257831 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ea80018e0 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7eb559a700 1 -- 172.21.15.114:0/1537257831 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f7ea8001c20 con 0x7f7eb81043b0 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.526+0000 7f7ebc862700 1 -- 172.21.15.114:0/1537257831 >> v1:172.21.15.114:6789/0 conn(0x7f7eb81043b0 legacy=0x7f7eb81047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:54.113 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.527+0000 7f7ebc862700 1 -- 172.21.15.114:0/1537257831 shutdown_connections 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.527+0000 7f7ebc862700 1 -- 172.21.15.114:0/1537257831 >> 172.21.15.114:0/1537257831 conn(0x7f7eb80fffd0 msgr2=0x7f7eb81023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.527+0000 7f7ebc862700 1 -- 172.21.15.114:0/1537257831 shutdown_connections 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.527+0000 7f7ebc862700 1 -- 172.21.15.114:0/1537257831 wait complete. 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.527+0000 7f7ebc862700 1 Processor -- start 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.527+0000 7f7ebc862700 1 -- start start 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ebc862700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7eb819b550 con 0x7f7eb81043b0 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7eb659c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7eb81043b0 0x7f7eb819ae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39112/0 (socket says 172.21.15.114:39112) 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7eb659c700 1 -- 172.21.15.114:0/2910942233 learned_addr learned my addr 172.21.15.114:0/2910942233 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498448749 0 0) 0x7f7eb819b550 con 0x7f7eb81043b0 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea00031f0 con 0x7f7eb81043b0 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012537281 0 0) 0x7f7ea00031f0 con 0x7f7eb81043b0 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7eb819b550 con 0x7f7eb81043b0 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f7ea8001540 con 0x7f7eb81043b0 2024-03-24T11:33:54.114 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1286833239 0 0) 0x7f7eb819b550 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb819b6e0 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb819b990 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb819f4e0 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ea8004c20 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f7ea8004f20 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7ea8005490 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.528+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7f7ea8006ae0 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.529+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb81948e0 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.531+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7ea8005920 con 0x7f7eb81043b0 2024-03-24T11:33:54.115 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.603+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7eb80632a0 con 0x7f7eb81043b0 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.604+0000 7f7ea77fe700 1 -- 172.21.15.114:0/2910942233 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (unknown 1155462804 0 2122390163) 0x7f7ea8005ac0 con 0x7f7eb81043b0 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.604+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 >> v1:172.21.15.114:6789/0 conn(0x7f7eb81043b0 legacy=0x7f7eb819ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.604+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 shutdown_connections 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.604+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 >> 172.21.15.114:0/2910942233 conn(0x7f7eb80fffd0 msgr2=0x7f7eb8191a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.605+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 shutdown_connections 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:53.605+0000 7f7ebc862700 1 -- 172.21.15.114:0/2910942233 wait complete. 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-03-24T11:33:54.116 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T11:33:55.166 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:33:55.166 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:33:55.166 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:55.166 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.114:6789] 2024-03-24T11:33:55.166 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:33:55.166 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:33:55.166 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.508+0000 7f3aa20ea700 1 Processor -- start 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.508+0000 7f3aa20ea700 1 -- start start 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3aa20ea700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3a9c079720 con 0x7f3a9c1065c0 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3a9b7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3a9c1065c0 0x7f3a9c1089b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39114/0 (socket says 172.21.15.114:39114) 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3a9b7fe700 1 -- 172.21.15.114:0/2631952208 learned_addr learned my addr 172.21.15.114:0/2631952208 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3a9a7fc700 1 -- 172.21.15.114:0/2631952208 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379661035 0 0) 0x7f3a9c079720 con 0x7f3a9c1065c0 2024-03-24T11:33:55.167 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3a9a7fc700 1 -- 172.21.15.114:0/2631952208 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a840035b0 con 0x7f3a9c1065c0 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3a9a7fc700 1 -- 172.21.15.114:0/2631952208 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 557087494 0 0) 0x7f3a840035b0 con 0x7f3a9c1065c0 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3a9a7fc700 1 -- 172.21.15.114:0/2631952208 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c10b0e0 con 0x7f3a9c1065c0 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.509+0000 7f3a9a7fc700 1 -- 172.21.15.114:0/2631952208 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f3a8c001580 con 0x7f3a9c1065c0 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.510+0000 7f3a9a7fc700 1 -- 172.21.15.114:0/2631952208 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a8c0018e0 con 0x7f3a9c1065c0 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.510+0000 7f3a9a7fc700 1 -- 172.21.15.114:0/2631952208 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f3a8c001c20 con 0x7f3a9c1065c0 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.510+0000 7f3aa20ea700 1 -- 172.21.15.114:0/2631952208 >> v1:172.21.15.114:6789/0 conn(0x7f3a9c1065c0 legacy=0x7f3a9c1089b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.510+0000 7f3aa20ea700 1 -- 172.21.15.114:0/2631952208 shutdown_connections 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.510+0000 7f3aa20ea700 1 -- 172.21.15.114:0/2631952208 >> 172.21.15.114:0/2631952208 conn(0x7f3a9c0fffd0 msgr2=0x7f3a9c1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.510+0000 7f3aa20ea700 1 -- 172.21.15.114:0/2631952208 shutdown_connections 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.510+0000 7f3aa20ea700 1 -- 172.21.15.114:0/2631952208 wait complete. 2024-03-24T11:33:55.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.511+0000 7f3aa20ea700 1 Processor -- start 2024-03-24T11:33:55.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.511+0000 7f3aa20ea700 1 -- start start 2024-03-24T11:33:55.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.511+0000 7f3aa20ea700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3a9c19ec90 con 0x7f3a9c1065c0 2024-03-24T11:33:55.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.511+0000 7f3a9b7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3a9c1065c0 0x7f3a9c105e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39120/0 (socket says 172.21.15.114:39120) 2024-03-24T11:33:55.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.511+0000 7f3a9b7fe700 1 -- 172.21.15.114:0/1324049233 learned_addr learned my addr 172.21.15.114:0/1324049233 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:55.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.511+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625959273 0 0) 0x7f3a9c19ec90 con 0x7f3a9c1065c0 2024-03-24T11:33:55.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a840031f0 con 0x7f3a9c1065c0 2024-03-24T11:33:55.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3620977541 0 0) 0x7f3a840031f0 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a9c19ec90 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f3a8c001480 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3643149597 0 0) 0x7f3a9c19ec90 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c19fe30 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9c19ee80 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9c19f380 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a8c004c80 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f3a8c004f00 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3a8c005470 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.512+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7f3a8c006a90 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.513+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a88005150 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.515+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3a8c005900 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.585+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3a88005980 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.587+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+318 (unknown 1213389831 0 4004080035) 0x7f3a8c005b60 con 0x7f3a9c1065c0 2024-03-24T11:33:55.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.588+0000 7f3a98ff9700 1 -- 172.21.15.114:0/1324049233 <== mon.0 v1:172.21.15.114:6789/0 11 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f3a8c007000 con 0x7f3a9c1065c0 2024-03-24T11:33:55.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.589+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 >> v1:172.21.15.114:6789/0 conn(0x7f3a9c1065c0 legacy=0x7f3a9c105e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:55.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.589+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 shutdown_connections 2024-03-24T11:33:55.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.589+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 >> 172.21.15.114:0/1324049233 conn(0x7f3a9c0fffd0 msgr2=0x7f3a9c18ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:55.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.589+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 shutdown_connections 2024-03-24T11:33:55.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:54.589+0000 7f3aa20ea700 1 -- 172.21.15.114:0/1324049233 wait complete. 2024-03-24T11:33:55.173 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-03-24T11:33:55.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:55 smithi114 ceph-mon[29825]: from='client.? v1:172.21.15.114:0/1324049233' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:33:55.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:55 smithi114 ceph-mon[29825]: from='client.? v1:172.21.15.114:0/1324049233' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:33:56.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:56 smithi114 ceph-mon[29825]: from='client.? v1:172.21.15.114:0/3780393002' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:56.286 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.573+0000 7fa8415e9700 1 Processor -- start 2024-03-24T11:33:56.290 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.573+0000 7fa8415e9700 1 -- start start 2024-03-24T11:33:56.290 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.573+0000 7fa8415e9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa83c109f20 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.573+0000 7fa83affd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa83c105af0 0x7fa83c105f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39124/0 (socket says 172.21.15.114:39124) 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.573+0000 7fa83affd700 1 -- 172.21.15.114:0/4217094624 learned_addr learned my addr 172.21.15.114:0/4217094624 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.573+0000 7fa83a7fc700 1 -- 172.21.15.114:0/4217094624 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344478412 0 0) 0x7fa83c109f20 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.573+0000 7fa83a7fc700 1 -- 172.21.15.114:0/4217094624 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8280035b0 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa83a7fc700 1 -- 172.21.15.114:0/4217094624 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590214484 0 0) 0x7fa8280035b0 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa83a7fc700 1 -- 172.21.15.114:0/4217094624 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c10b0c0 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa83a7fc700 1 -- 172.21.15.114:0/4217094624 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa82c001580 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa83a7fc700 1 -- 172.21.15.114:0/4217094624 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fa82c001ce0 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa83a7fc700 1 -- 172.21.15.114:0/4217094624 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa82c005960 con 0x7fa83c105af0 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa8415e9700 1 -- 172.21.15.114:0/4217094624 >> v1:172.21.15.114:6789/0 conn(0x7fa83c105af0 legacy=0x7fa83c105f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa8415e9700 1 -- 172.21.15.114:0/4217094624 shutdown_connections 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.574+0000 7fa8415e9700 1 -- 172.21.15.114:0/4217094624 >> 172.21.15.114:0/4217094624 conn(0x7fa83c101070 msgr2=0x7fa83c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.575+0000 7fa8415e9700 1 -- 172.21.15.114:0/4217094624 shutdown_connections 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.575+0000 7fa8415e9700 1 -- 172.21.15.114:0/4217094624 wait complete. 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.575+0000 7fa8415e9700 1 Processor -- start 2024-03-24T11:33:56.291 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.575+0000 7fa8415e9700 1 -- start start 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.575+0000 7fa8415e9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa83c199ae0 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.575+0000 7fa83affd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa83c105af0 0x7fa83c1993d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39130/0 (socket says 172.21.15.114:39130) 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.575+0000 7fa83affd700 1 -- 172.21.15.114:0/3780393002 learned_addr learned my addr 172.21.15.114:0/3780393002 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892922869 0 0) 0x7fa83c199ae0 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8280054f0 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3807755829 0 0) 0x7fa8280054f0 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa83c199ae0 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa82c005940 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1749237706 0 0) 0x7fa83c199ae0 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c199c70 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83c199f20 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83c19da70 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fa82c001b60 con 0x7fa83c105af0 2024-03-24T11:33:56.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa82c0061c0 con 0x7fa83c105af0 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa82c006790 con 0x7fa83c105af0 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.576+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7fa82c007e10 con 0x7fa83c105af0 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.577+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c005150 con 0x7fa83c105af0 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.579+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa82c005b60 con 0x7fa83c105af0 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.646+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa81c005400 con 0x7fa83c105af0 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.647+0000 7fa838ff9700 1 -- 172.21.15.114:0/3780393002 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+148 (unknown 1452402520 0 2046223321) 0x7fa82c001080 con 0x7fa83c105af0 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.647+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 >> v1:172.21.15.114:6789/0 conn(0x7fa83c105af0 legacy=0x7fa83c1993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.647+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 shutdown_connections 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.647+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 >> 172.21.15.114:0/3780393002 conn(0x7fa83c101070 msgr2=0x7fa83c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.647+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 shutdown_connections 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:55.647+0000 7fa8415e9700 1 -- 172.21.15.114:0/3780393002 wait complete. 2024-03-24T11:33:56.293 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-03-24T11:33:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:56 smithi114 systemd[1]: Stopping Ceph mon.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:33:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:56 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a[29802]: 2024-03-24T11:33:56.530+0000 7fc861482700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:33:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:56 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a[29802]: 2024-03-24T11:33:56.530+0000 7fc861482700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:33:56.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:56 smithi114 podman[30155]: 2024-03-24 11:33:56.736000536 +0000 UTC m=+0.290449436 container died b0dd44c094815a10aead92c0fbba78277232460454e417c5df1107629fb1d14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T11:33:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:57 smithi114 podman[30155]: 2024-03-24 11:33:57.377892566 +0000 UTC m=+0.932341464 container cleanup b0dd44c094815a10aead92c0fbba78277232460454e417c5df1107629fb1d14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T11:33:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:57 smithi114 bash[30155]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:33:58.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 podman[30172]: 2024-03-24 11:33:58.004367656 +0000 UTC m=+1.268426043 container remove b0dd44c094815a10aead92c0fbba78277232460454e417c5df1107629fb1d14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-24T11:33:58.562 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T11:33:58.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 systemd[1]: Stopped Ceph mon.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:33:58.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 systemd[1]: Starting Ceph mon.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:33:58.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 podman[30273]: 2024-03-24 11:33:58.446964936 +0000 UTC m=+0.093626303 container create a9e456820c15a3664054c80dea0a630340377bb17179435f2ebaaf2d489b9b63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T11:33:58.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 podman[30273]: 2024-03-24 11:33:58.371417763 +0000 UTC m=+0.018079136 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:33:58.728 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 podman[30273]: 2024-03-24 11:33:58.641108146 +0000 UTC m=+0.287769525 container init a9e456820c15a3664054c80dea0a630340377bb17179435f2ebaaf2d489b9b63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 podman[30273]: 2024-03-24 11:33:58.647037746 +0000 UTC m=+0.293699122 container start a9e456820c15a3664054c80dea0a630340377bb17179435f2ebaaf2d489b9b63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: pidfile_write: ignore empty --pid-file 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: load: jerasure load: lrc 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Git sha 0 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: DB SUMMARY 2024-03-24T11:33:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: DB Session ID: 9KBCBAM5JKKE9TIT24T1 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 85572 ; 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.env: 0x5637935c6ae0 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:33:58.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.info_log: 0x563794cbd900 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.statistics: (nil) 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.use_fsync: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.db_log_dir: 2024-03-24T11:33:58.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.wal_dir: 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.write_buffer_manager: 0x563794c91400 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:33:58.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.unordered_write: 0 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.row_cache: None 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.wal_filter: None 2024-03-24T11:33:58.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:33:58.964 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:33:58.964 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.wal_compression: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:33:58.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_open_files: -1 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:33:58.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Compression algorithms supported: 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kZSTD supported: 0 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.merge_operator: 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_filter: None 2024-03-24T11:33:58.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:33:58.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:33:58.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:33:58.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:33:58.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563794cbda00) 2024-03-24T11:33:58.968 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:33:58.969 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:33:58.970 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:33:58.970 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:33:58.970 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_type: 0 2024-03-24T11:33:58.970 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_index_type: 0 2024-03-24T11:33:58.970 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_shortening: 1 2024-03-24T11:33:58.970 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:33:58.970 INFO:journalctl@ceph.mon.a.smithi114.stdout: checksum: 4 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: no_block_cache: 0 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache: 0x563794cc1090 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_options: 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: capacity : 536870912 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_shard_bits : 4 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: strict_capacity_limit : 0 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_compressed: (nil) 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: persistent_cache: (nil) 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size: 4096 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size_deviation: 10 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_restart_interval: 16 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_block_restart_interval: 1 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_block_size: 4096 2024-03-24T11:33:58.971 INFO:journalctl@ceph.mon.a.smithi114.stdout: partition_filters: 0 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: use_delta_encoding: 1 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: filter_policy: bloomfilter 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: whole_key_filtering: 1 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: verify_compression: 0 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: format_version: 5 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_index_compression: 1 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_align: 0 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_auto_readahead_size: 262144 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: prepopulate_block_cache: 0 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression: NoCompression 2024-03-24T11:33:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.num_levels: 7 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:33:58.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:33:58.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:33:58.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:33:58.976 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.ttl: 2592000 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:33:58.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 137aab5f-8f5d-4044-b4ad-48952e6861a7 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280038676468, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280038677317, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81916, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 264, "table_properties": {"data_size": 80110, "index_size": 216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10703, "raw_average_key_size": 47, "raw_value_size": 74080, "raw_average_value_size": 332, "num_data_blocks": 10, "num_entries": 223, "num_filter_entries": 223, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280038, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "137aab5f-8f5d-4044-b4ad-48952e6861a7", "db_session_id": "9KBCBAM5JKKE9TIT24T1", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280038677374, "job": 1, "event": "recovery_finished"} 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:33:58.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563794db2000 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: DB pointer 0x563794d9c000 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** DB Stats ** 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:33:58.979 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 2/0 81.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 81.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:33:58.980 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 28.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 28.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache BinnedLRUCache@0x563794cc1090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: starting mon.a rank 0 at public addrs v1:172.21.15.114:6789/0 at bind addrs v1:172.21.15.114:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:58.981 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???) e1 preinit fsid 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???).mds e1 new map 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???).mds e1 print_map 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout: e1 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout: legacy client fscid: -1 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout: No filesystems configured 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 bash[30273]: a9e456820c15a3664054c80dea0a630340377bb17179435f2ebaaf2d489b9b63 2024-03-24T11:33:58.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 systemd[1]: Started Ceph mon.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:33:58.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:33:58.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: monmap e1: 1 mons at {a=v1:172.21.15.114:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:33:58.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: fsmap 2024-03-24T11:33:58.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:33:58.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:33:58 smithi114 ceph-mon[30312]: mgrmap e1: no daemons active 2024-03-24T11:33:59.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.189+0000 7f0a15a8f700 1 Processor -- start 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.189+0000 7f0a15a8f700 1 -- start start 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.189+0000 7f0a15a8f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a10108e20 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0f7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0a101049f0 0x7f0a10104e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39134/0 (socket says 172.21.15.114:39134) 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0f7fe700 1 -- 172.21.15.114:0/1345117521 learned_addr learned my addr 172.21.15.114:0/1345117521 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0e7fc700 1 -- 172.21.15.114:0/1345117521 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117723509 0 0) 0x7f0a10108e20 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0e7fc700 1 -- 172.21.15.114:0/1345117521 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f80035b0 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0e7fc700 1 -- 172.21.15.114:0/1345117521 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447988339 0 0) 0x7f09f80035b0 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0e7fc700 1 -- 172.21.15.114:0/1345117521 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10109fc0 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0e7fc700 1 -- 172.21.15.114:0/1345117521 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a00001900 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0e7fc700 1 -- 172.21.15.114:0/1345117521 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f0a00002060 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.190+0000 7f0a0e7fc700 1 -- 172.21.15.114:0/1345117521 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a00005ce0 con 0x7f0a101049f0 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.191+0000 7f0a15a8f700 1 -- 172.21.15.114:0/1345117521 >> v1:172.21.15.114:6789/0 conn(0x7f0a101049f0 legacy=0x7f0a10104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.191+0000 7f0a15a8f700 1 -- 172.21.15.114:0/1345117521 shutdown_connections 2024-03-24T11:33:59.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.191+0000 7f0a15a8f700 1 -- 172.21.15.114:0/1345117521 >> 172.21.15.114:0/1345117521 conn(0x7f0a100fff70 msgr2=0x7f0a101023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.191+0000 7f0a15a8f700 1 -- 172.21.15.114:0/1345117521 shutdown_connections 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.191+0000 7f0a15a8f700 1 -- 172.21.15.114:0/1345117521 wait complete. 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a15a8f700 1 Processor -- start 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a15a8f700 1 -- start start 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a15a8f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a10079300 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a0f7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0a101049f0 0x7f0a1019c170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39140/0 (socket says 172.21.15.114:39140) 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a0f7fe700 1 -- 172.21.15.114:0/3419526365 learned_addr learned my addr 172.21.15.114:0/3419526365 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267007438 0 0) 0x7f0a10079300 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f80054f0 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3044912528 0 0) 0x7f09f80054f0 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a10079300 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.192+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a00005cc0 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4138595391 0 0) 0x7f0a10079300 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10079490 con 0x7f0a101049f0 2024-03-24T11:33:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a10079740 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1019dae0 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f0a00001ea0 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a00006550 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0a00006b20 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.193+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7f0a00008100 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.194+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc005150 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.196+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0a00005a10 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.264+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 --> v1:172.21.15.114:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f09fc005980 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.266+0000 7f0a0cff9700 1 -- 172.21.15.114:0/3419526365 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (unknown 534127980 0 0) 0x7f0a00004d50 con 0x7f0a101049f0 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.267+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 >> v1:172.21.15.114:6789/0 conn(0x7f0a101049f0 legacy=0x7f0a1019c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.267+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 shutdown_connections 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.267+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 >> 172.21.15.114:0/3419526365 conn(0x7f0a100fff70 msgr2=0x7f0a101023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.267+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 shutdown_connections 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:33:59.267+0000 7f0a15a8f700 1 -- 172.21.15.114:0/3419526365 wait complete. 2024-03-24T11:33:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:33:59.805 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:33:59.805 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-03-24T11:33:59.805 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T11:33:59.806 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T11:34:00.051 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:34:00.051 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T11:34:00.203 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-412c996a-e9d2-11ee-95cd-87774f69a715.target.wants/ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-412c996a-e9d2-11ee-95cd-87774f69a715@.service. 2024-03-24T11:34:00.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3419526365' entity='client.admin' 2024-03-24T11:34:00.489 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:00 smithi114 podman[30555]: 2024-03-24 11:34:00.441095604 +0000 UTC m=+0.120130672 container create f3074bd1a81c5e973a15fd36a98e2b34d298b155d424b5287d0221f714cbe98c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T11:34:00.782 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:34:00.782 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-03-24T11:34:00.790 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:34:00.790 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-03-24T11:34:00.790 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-03-24T11:34:00.790 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:34:00.799 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:34:00.799 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-03-24T11:34:00.806 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:34:00.807 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-03-24T11:34:00.807 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-03-24T11:34:00.807 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T11:34:00.807 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-03-24T11:34:00.807 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-03-24T11:34:00.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:00 smithi114 podman[30555]: 2024-03-24 11:34:00.652013355 +0000 UTC m=+0.331048431 container init f3074bd1a81c5e973a15fd36a98e2b34d298b155d424b5287d0221f714cbe98c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T11:34:00.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:00 smithi114 podman[30555]: 2024-03-24 11:34:00.657592657 +0000 UTC m=+0.336627724 container start f3074bd1a81c5e973a15fd36a98e2b34d298b155d424b5287d0221f714cbe98c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T11:34:00.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:00 smithi114 bash[30555]: f3074bd1a81c5e973a15fd36a98e2b34d298b155d424b5287d0221f714cbe98c 2024-03-24T11:34:00.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:00 smithi114 systemd[1]: Started Ceph mgr.y for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:34:01.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3328109660' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:34:01.489 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:01 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:01.210+0000 7f648dd3f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:34:01.489 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:01 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:01.307+0000 7f648dd3f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:34:01.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:34:01.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:01.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "412c996a-e9d2-11ee-95cd-87774f69a715", 2024-03-24T11:34:01.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:34:01.888 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:34:01.889 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:01.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:34:01.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:01.890 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:34:01.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:34:01.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:01.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:34:01.899 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:33:52.998488+0000", 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.265+0000 7f4a65545700 1 Processor -- start 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.265+0000 7f4a65545700 1 -- start start 2024-03-24T11:34:01.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.265+0000 7f4a65545700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a60108e20 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5effd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4a601049f0 0x7f4a60104e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36560/0 (socket says 172.21.15.114:36560) 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5effd700 1 -- 172.21.15.114:0/2495024962 learned_addr learned my addr 172.21.15.114:0/2495024962 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5dffb700 1 -- 172.21.15.114:0/2495024962 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849830338 0 0) 0x7f4a60108e20 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5dffb700 1 -- 172.21.15.114:0/2495024962 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a480057c0 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5dffb700 1 -- 172.21.15.114:0/2495024962 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3328716999 0 0) 0x7f4a480057c0 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5dffb700 1 -- 172.21.15.114:0/2495024962 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60109fc0 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5dffb700 1 -- 172.21.15.114:0/2495024962 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4a50001580 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5dffb700 1 -- 172.21.15.114:0/2495024962 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f4a50001ce0 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.266+0000 7f4a5dffb700 1 -- 172.21.15.114:0/2495024962 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4a50005960 con 0x7f4a601049f0 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.267+0000 7f4a65545700 1 -- 172.21.15.114:0/2495024962 >> v1:172.21.15.114:6789/0 conn(0x7f4a601049f0 legacy=0x7f4a60104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.267+0000 7f4a65545700 1 -- 172.21.15.114:0/2495024962 shutdown_connections 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.267+0000 7f4a65545700 1 -- 172.21.15.114:0/2495024962 >> 172.21.15.114:0/2495024962 conn(0x7f4a600fff70 msgr2=0x7f4a601023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.267+0000 7f4a65545700 1 -- 172.21.15.114:0/2495024962 shutdown_connections 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.267+0000 7f4a65545700 1 -- 172.21.15.114:0/2495024962 wait complete. 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.268+0000 7f4a65545700 1 Processor -- start 2024-03-24T11:34:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.268+0000 7f4a65545700 1 -- start start 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.268+0000 7f4a65545700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a60197900 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.268+0000 7f4a5effd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4a601049f0 0x7f4a601971f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36576/0 (socket says 172.21.15.114:36576) 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.268+0000 7f4a5effd700 1 -- 172.21.15.114:0/3328109660 learned_addr learned my addr 172.21.15.114:0/3328109660 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728824903 0 0) 0x7f4a60197900 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a48007700 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1686393547 0 0) 0x7f4a48007700 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a60197900 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4a50005940 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3742522026 0 0) 0x7f4a60197900 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60197a90 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a60197d40 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6019b890 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f4a50001b20 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4a500060b0 con 0x7f4a601049f0 2024-03-24T11:34:01.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.269+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4a500066a0 con 0x7f4a601049f0 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.270+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7f4a50007cb0 con 0x7f4a601049f0 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.270+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c005150 con 0x7f4a601049f0 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.272+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4a50005690 con 0x7f4a601049f0 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.344+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4a4c005400 con 0x7f4a601049f0 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.344+0000 7f4a47fff700 1 -- 172.21.15.114:0/3328109660 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 1623859272) 0x7f4a50001110 con 0x7f4a601049f0 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.345+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 >> v1:172.21.15.114:6789/0 conn(0x7f4a601049f0 legacy=0x7f4a601971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.345+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 shutdown_connections 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.345+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 >> 172.21.15.114:0/3328109660 conn(0x7f4a600fff70 msgr2=0x7f4a601023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.345+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 shutdown_connections 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:01.345+0000 7f4a65545700 1 -- 172.21.15.114:0/3328109660 wait complete. 2024-03-24T11:34:01.903 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-03-24T11:34:02.238 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:01 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:01.961+0000 7f648dd3f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:34:02.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.054+0000 7f648dd3f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:34:02.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.192+0000 7f648dd3f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:34:02.625 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.285+0000 7f648dd3f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:34:02.625 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.366+0000 7f648dd3f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:34:02.625 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.624+0000 7f648dd3f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:34:02.906 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.731+0000 7f648dd3f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:02.906 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.813+0000 7f648dd3f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:34:02.906 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:02 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:02.905+0000 7f648dd3f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:34:03.238 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:03 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:03.081+0000 7f648dd3f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:34:04.311 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:04.050+0000 7f648dd3f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:34:04.312 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:04.184+0000 7f648dd3f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:34:04.636 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2492724775' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:34:04.636 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:04.546+0000 7f648dd3f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:34:04.636 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:04.635+0000 7f648dd3f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:34:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:04.832+0000 7f648dd3f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "412c996a-e9d2-11ee-95cd-87774f69a715", 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:34:05.044 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:05.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:34:05.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:34:05.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:33:52.998488+0000", 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc086ad4700 1 Processor -- start 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc086ad4700 1 -- start start 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc086ad4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc08010a030 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc084870700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc080103e60 0x7fc080104280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36580/0 (socket says 172.21.15.114:36580) 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc084870700 1 -- 172.21.15.114:0/445203297 learned_addr learned my addr 172.21.15.114:0/445203297 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc07f7fe700 1 -- 172.21.15.114:0/445203297 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508633451 0 0) 0x7fc08010a030 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc07f7fe700 1 -- 172.21.15.114:0/445203297 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0700035b0 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.415+0000 7fc07f7fe700 1 -- 172.21.15.114:0/445203297 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1871642514 0 0) 0x7fc0700035b0 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc07f7fe700 1 -- 172.21.15.114:0/445203297 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08010b1d0 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc07f7fe700 1 -- 172.21.15.114:0/445203297 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fc074001580 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc07f7fe700 1 -- 172.21.15.114:0/445203297 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fc074001ce0 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc07f7fe700 1 -- 172.21.15.114:0/445203297 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fc074005960 con 0x7fc080103e60 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc086ad4700 1 -- 172.21.15.114:0/445203297 >> v1:172.21.15.114:6789/0 conn(0x7fc080103e60 legacy=0x7fc080104280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc086ad4700 1 -- 172.21.15.114:0/445203297 shutdown_connections 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc086ad4700 1 -- 172.21.15.114:0/445203297 >> 172.21.15.114:0/445203297 conn(0x7fc0800ff700 msgr2=0x7fc080101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc086ad4700 1 -- 172.21.15.114:0/445203297 shutdown_connections 2024-03-24T11:34:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.416+0000 7fc086ad4700 1 -- 172.21.15.114:0/445203297 wait complete. 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.417+0000 7fc086ad4700 1 Processor -- start 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.417+0000 7fc086ad4700 1 -- start start 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.417+0000 7fc086ad4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc080199b60 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.417+0000 7fc084870700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc080103e60 0x7fc080199450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36596/0 (socket says 172.21.15.114:36596) 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.417+0000 7fc084870700 1 -- 172.21.15.114:0/2492724775 learned_addr learned my addr 172.21.15.114:0/2492724775 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.417+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641735642 0 0) 0x7fc080199b60 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0700054f0 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3723593277 0 0) 0x7fc0700054f0 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc080199b60 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fc074001080 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4248031301 0 0) 0x7fc080199b60 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080199cf0 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc080199fa0 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fc074005970 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fc074006290 con 0x7fc080103e60 2024-03-24T11:34:05.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08019daf0 con 0x7fc080103e60 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc0740068c0 con 0x7fc080103e60 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.418+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7fc074006eb0 con 0x7fc080103e60 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.419+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08010acc0 con 0x7fc080103e60 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.421+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc074001870 con 0x7fc080103e60 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.494+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc080192b00 con 0x7fc080103e60 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.494+0000 7fc07dffb700 1 -- 172.21.15.114:0/2492724775 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 256568054) 0x7fc074005bf0 con 0x7fc080103e60 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.495+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 >> v1:172.21.15.114:6789/0 conn(0x7fc080103e60 legacy=0x7fc080199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.495+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 shutdown_connections 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.495+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 >> 172.21.15.114:0/2492724775 conn(0x7fc0800ff700 msgr2=0x7fc080101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.495+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 shutdown_connections 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:04.496+0000 7fc086ad4700 1 -- 172.21.15.114:0/2492724775 wait complete. 2024-03-24T11:34:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (2/15)... 2024-03-24T11:34:05.289 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:05 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:05.014+0000 7f648dd3f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:34:05.543 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:05 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:05.289+0000 7f648dd3f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:05.543 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:05 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:05.542+0000 7f648dd3f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:05.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:05 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:05.775+0000 7f648dd3f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:34:06.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:06 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:06.571+0000 7f648dd3f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:34:07.400 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:07.141+0000 7f648dd3f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:34:07.400 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:07.239+0000 7f648dd3f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:07.459+0000 7f648dd3f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: Activating manager daemon y 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: mgrmap e2: y(active, starting, since 0.00302017s) 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:34:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: Manager daemon y is now available 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' 2024-03-24T11:34:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:07 smithi114 ceph-mon[30312]: from='mgr.14100 v1:172.21.15.114:0/336527050' entity='mgr.y' 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "412c996a-e9d2-11ee-95cd-87774f69a715", 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:34:08.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:34:08.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:34:08.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:34:08.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:08.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:33:52.998488+0000", 2024-03-24T11:34:08.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.502+0000 7f4b29485700 1 Processor -- start 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.502+0000 7f4b29485700 1 -- start start 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.502+0000 7f4b29485700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b24109f40 con 0x7f4b24105b10 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.502+0000 7f4b22ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4b24105b10 0x7f4b24105f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36650/0 (socket says 172.21.15.114:36650) 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.502+0000 7f4b22ffd700 1 -- 172.21.15.114:0/734100732 learned_addr learned my addr 172.21.15.114:0/734100732 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.502+0000 7f4b21ffb700 1 -- 172.21.15.114:0/734100732 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127014163 0 0) 0x7f4b24109f40 con 0x7f4b24105b10 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.502+0000 7f4b21ffb700 1 -- 172.21.15.114:0/734100732 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c0035b0 con 0x7f4b24105b10 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b21ffb700 1 -- 172.21.15.114:0/734100732 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520629559 0 0) 0x7f4b0c0035b0 con 0x7f4b24105b10 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b21ffb700 1 -- 172.21.15.114:0/734100732 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2410b0e0 con 0x7f4b24105b10 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b21ffb700 1 -- 172.21.15.114:0/734100732 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4b14001580 con 0x7f4b24105b10 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b21ffb700 1 -- 172.21.15.114:0/734100732 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f4b14001ce0 con 0x7f4b24105b10 2024-03-24T11:34:08.096 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b21ffb700 1 -- 172.21.15.114:0/734100732 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4b14005960 con 0x7f4b24105b10 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b29485700 1 -- 172.21.15.114:0/734100732 >> v1:172.21.15.114:6789/0 conn(0x7f4b24105b10 legacy=0x7f4b24105f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b29485700 1 -- 172.21.15.114:0/734100732 shutdown_connections 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b29485700 1 -- 172.21.15.114:0/734100732 >> 172.21.15.114:0/734100732 conn(0x7f4b241010d0 msgr2=0x7f4b241034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b29485700 1 -- 172.21.15.114:0/734100732 shutdown_connections 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.503+0000 7f4b29485700 1 -- 172.21.15.114:0/734100732 wait complete. 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.504+0000 7f4b29485700 1 Processor -- start 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.504+0000 7f4b29485700 1 -- start start 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.504+0000 7f4b29485700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b24199b50 con 0x7f4b24105b10 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b22ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4b24105b10 0x7f4b24199440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36656/0 (socket says 172.21.15.114:36656) 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b22ffd700 1 -- 172.21.15.114:0/700126577 learned_addr learned my addr 172.21.15.114:0/700126577 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880021681 0 0) 0x7f4b24199b50 con 0x7f4b24105b10 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c0054f0 con 0x7f4b24105b10 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 640109482 0 0) 0x7f4b0c0054f0 con 0x7f4b24105b10 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b24199b50 con 0x7f4b24105b10 2024-03-24T11:34:08.097 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4b14005940 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3639237146 0 0) 0x7f4b24199b50 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24199ce0 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b24199f90 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.505+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2419dae0 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.506+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f4b14001b20 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.506+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4b140060b0 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.506+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 2) v1 ==== 45525+0+0 (unknown 524329404 0 0) 0x7f4b14011410 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.507+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7f4b14047960 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.507+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b10005150 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.509+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4b14016900 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.582+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4b10005400 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.582+0000 7f4b0bfff700 1 -- 172.21.15.114:0/700126577 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 2172940599) 0x7f4b14014cd0 con 0x7f4b24105b10 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.583+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 >> v1:172.21.15.114:6789/0 conn(0x7f4b24105b10 legacy=0x7f4b24199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:08.098 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.583+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 shutdown_connections 2024-03-24T11:34:08.099 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.583+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 >> 172.21.15.114:0/700126577 conn(0x7f4b241010d0 msgr2=0x7f4b241034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:08.099 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.583+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 shutdown_connections 2024-03-24T11:34:08.099 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:07.583+0000 7f4b29485700 1 -- 172.21.15.114:0/700126577 wait complete. 2024-03-24T11:34:08.099 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (3/15)... 2024-03-24T11:34:08.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/700126577' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:34:08.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:08 smithi114 ceph-mon[30312]: mgrmap e3: y(active, since 1.0074s) 2024-03-24T11:34:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:10 smithi114 ceph-mon[30312]: mgrmap e4: y(active, since 2s) 2024-03-24T11:34:11.332 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:34:11.332 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:11.332 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "412c996a-e9d2-11ee-95cd-87774f69a715", 2024-03-24T11:34:11.332 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:34:11.332 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:34:11.332 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:34:11.333 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:11.334 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:34:11.334 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:11.334 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:34:11.334 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:34:11.334 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:34:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:34:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:34:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:34:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:34:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:33:52.998488+0000", 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.505+0000 7ffbd6543700 1 Processor -- start 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.505+0000 7ffbd6543700 1 -- start start 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.505+0000 7ffbd6543700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffbd0079760 con 0x7ffbd0106580 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbcffff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffbd0106580 0x7ffbd0108970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36694/0 (socket says 172.21.15.114:36694) 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbcffff700 1 -- 172.21.15.114:0/243675815 learned_addr learned my addr 172.21.15.114:0/243675815 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbceffd700 1 -- 172.21.15.114:0/243675815 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384398741 0 0) 0x7ffbd0079760 con 0x7ffbd0106580 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbceffd700 1 -- 172.21.15.114:0/243675815 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb80035b0 con 0x7ffbd0106580 2024-03-24T11:34:11.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbceffd700 1 -- 172.21.15.114:0/243675815 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248200611 0 0) 0x7ffbb80035b0 con 0x7ffbd0106580 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbceffd700 1 -- 172.21.15.114:0/243675815 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd010b0a0 con 0x7ffbd0106580 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbceffd700 1 -- 172.21.15.114:0/243675815 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ffbc0001580 con 0x7ffbd0106580 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbceffd700 1 -- 172.21.15.114:0/243675815 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7ffbc0001ce0 con 0x7ffbd0106580 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.506+0000 7ffbceffd700 1 -- 172.21.15.114:0/243675815 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ffbc0005960 con 0x7ffbd0106580 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.507+0000 7ffbd6543700 1 -- 172.21.15.114:0/243675815 >> v1:172.21.15.114:6789/0 conn(0x7ffbd0106580 legacy=0x7ffbd0108970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.507+0000 7ffbd6543700 1 -- 172.21.15.114:0/243675815 shutdown_connections 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.507+0000 7ffbd6543700 1 -- 172.21.15.114:0/243675815 >> 172.21.15.114:0/243675815 conn(0x7ffbd00fff70 msgr2=0x7ffbd01023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.507+0000 7ffbd6543700 1 -- 172.21.15.114:0/243675815 shutdown_connections 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.507+0000 7ffbd6543700 1 -- 172.21.15.114:0/243675815 wait complete. 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.507+0000 7ffbd6543700 1 Processor -- start 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.507+0000 7ffbd6543700 1 -- start start 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbd6543700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffbd0199b40 con 0x7ffbd0106580 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcffff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffbd0106580 0x7ffbd0199430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36706/0 (socket says 172.21.15.114:36706) 2024-03-24T11:34:11.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcffff700 1 -- 172.21.15.114:0/2561951577 learned_addr learned my addr 172.21.15.114:0/2561951577 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499456960 0 0) 0x7ffbd0199b40 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb80054f0 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 837420843 0 0) 0x7ffbb80054f0 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbd0199b40 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ffbc0004870 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1184497115 0 0) 0x7ffbd0199b40 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd0199cd0 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.508+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd0199f80 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.509+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd019dad0 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.509+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7ffbc0004a40 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.509+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ffbc0006160 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.509+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 45723+0+0 (unknown 2752121932 0 0) 0x7ffbc0006430 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.509+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc005150 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.510+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7ffbc0046fd0 con 0x7ffbd0106580 2024-03-24T11:34:11.346 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.514+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ffbc0006c00 con 0x7ffbd0106580 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.783+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ffbbc005400 con 0x7ffbd0106580 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.783+0000 7ffbcd7fa700 1 -- 172.21.15.114:0/2561951577 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 857055345) 0x7ffbc0014f60 con 0x7ffbd0106580 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.785+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 >> v1:172.21.15.114:6800/1606950246 conn(0x7ffbb8039eb0 legacy=0x7ffbb803c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.785+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 >> v1:172.21.15.114:6789/0 conn(0x7ffbd0106580 legacy=0x7ffbd0199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.785+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 shutdown_connections 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.785+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 >> 172.21.15.114:0/2561951577 conn(0x7ffbd00fff70 msgr2=0x7ffbd0190260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.785+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 shutdown_connections 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:10.785+0000 7ffbd6543700 1 -- 172.21.15.114:0/2561951577 wait complete. 2024-03-24T11:34:11.347 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-03-24T11:34:11.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2561951577' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:34:12.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:34:12.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:34:12.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:34:12.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.748+0000 7fa268273700 1 Processor -- start 2024-03-24T11:34:12.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.748+0000 7fa268273700 1 -- start start 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.748+0000 7fa268273700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa260108e40 con 0x7fa260104a10 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.748+0000 7fa26600f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa260104a10 0x7fa260104e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48334/0 (socket says 172.21.15.114:48334) 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.748+0000 7fa26600f700 1 -- 172.21.15.114:0/2540498593 learned_addr learned my addr 172.21.15.114:0/2540498593 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.748+0000 7fa26500d700 1 -- 172.21.15.114:0/2540498593 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293152341 0 0) 0x7fa260108e40 con 0x7fa260104a10 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.748+0000 7fa26500d700 1 -- 172.21.15.114:0/2540498593 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa24c0035b0 con 0x7fa260104a10 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa26500d700 1 -- 172.21.15.114:0/2540498593 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3254378580 0 0) 0x7fa24c0035b0 con 0x7fa260104a10 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa26500d700 1 -- 172.21.15.114:0/2540498593 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa260109fe0 con 0x7fa260104a10 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa26500d700 1 -- 172.21.15.114:0/2540498593 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa25c001580 con 0x7fa260104a10 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa26500d700 1 -- 172.21.15.114:0/2540498593 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fa25c001ce0 con 0x7fa260104a10 2024-03-24T11:34:12.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa26500d700 1 -- 172.21.15.114:0/2540498593 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa25c005960 con 0x7fa260104a10 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa268273700 1 -- 172.21.15.114:0/2540498593 >> v1:172.21.15.114:6789/0 conn(0x7fa260104a10 legacy=0x7fa260104e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa268273700 1 -- 172.21.15.114:0/2540498593 shutdown_connections 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa268273700 1 -- 172.21.15.114:0/2540498593 >> 172.21.15.114:0/2540498593 conn(0x7fa2600fffd0 msgr2=0x7fa2601023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa268273700 1 -- 172.21.15.114:0/2540498593 shutdown_connections 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.749+0000 7fa268273700 1 -- 172.21.15.114:0/2540498593 wait complete. 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.750+0000 7fa268273700 1 Processor -- start 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.750+0000 7fa268273700 1 -- start start 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.750+0000 7fa268273700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa260079320 con 0x7fa260104a10 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.750+0000 7fa26600f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa260104a10 0x7fa26019c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48344/0 (socket says 172.21.15.114:48344) 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.750+0000 7fa26600f700 1 -- 172.21.15.114:0/3935327607 learned_addr learned my addr 172.21.15.114:0/3935327607 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384422928 0 0) 0x7fa260079320 con 0x7fa260104a10 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa24c0054f0 con 0x7fa260104a10 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2171674622 0 0) 0x7fa24c0054f0 con 0x7fa260104a10 2024-03-24T11:34:12.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa260079320 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa25c005940 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1898997481 0 0) 0x7fa260079320 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2600794b0 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa260079760 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26019dab0 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fa25c001b20 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.751+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa25c0060b0 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.752+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 45723+0+0 (unknown 2752121932 0 0) 0x7fa25c0114c0 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.752+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa244005150 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.752+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7fa25c046f20 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.757+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fa25c012cd0 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.949+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa244005980 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.950+0000 7fa2577fe700 1 -- 172.21.15.114:0/3935327607 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (unknown 1187553405 0 4238065447) 0x7fa25c014dd0 con 0x7fa260104a10 2024-03-24T11:34:12.505 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.952+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 >> v1:172.21.15.114:6800/1606950246 conn(0x7fa24c039f40 legacy=0x7fa24c03c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:12.506 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.952+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 >> v1:172.21.15.114:6789/0 conn(0x7fa260104a10 legacy=0x7fa26019c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:12.506 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.952+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 shutdown_connections 2024-03-24T11:34:12.506 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.952+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 >> 172.21.15.114:0/3935327607 conn(0x7fa2600fffd0 msgr2=0x7fa2601023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:12.506 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.952+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 shutdown_connections 2024-03-24T11:34:12.506 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:11.952+0000 7fa268273700 1 -- 172.21.15.114:0/3935327607 wait complete. 2024-03-24T11:34:12.506 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-03-24T11:34:12.845 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3935327607' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:34:13.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4054313714' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:34:13.858 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:13 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ignoring --setuser ceph since I am not root 2024-03-24T11:34:13.858 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:13 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ignoring --setgroup ceph since I am not root 2024-03-24T11:34:14.116 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:14.089+0000 7fda653d6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:34:14.122 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.949+0000 7fce5a195700 1 Processor -- start 2024-03-24T11:34:14.122 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.949+0000 7fce5a195700 1 -- start start 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.949+0000 7fce5a195700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce54079720 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.949+0000 7fce537fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fce541065a0 0x7fce54108990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48352/0 (socket says 172.21.15.114:48352) 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.949+0000 7fce537fe700 1 -- 172.21.15.114:0/3996666228 learned_addr learned my addr 172.21.15.114:0/3996666228 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.949+0000 7fce527fc700 1 -- 172.21.15.114:0/3996666228 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378523907 0 0) 0x7fce54079720 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.949+0000 7fce527fc700 1 -- 172.21.15.114:0/3996666228 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce400035b0 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce527fc700 1 -- 172.21.15.114:0/3996666228 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787117811 0 0) 0x7fce400035b0 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce527fc700 1 -- 172.21.15.114:0/3996666228 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5410b0c0 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce527fc700 1 -- 172.21.15.114:0/3996666228 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fce3c001580 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce527fc700 1 -- 172.21.15.114:0/3996666228 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fce3c001ce0 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce527fc700 1 -- 172.21.15.114:0/3996666228 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fce3c005960 con 0x7fce541065a0 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce5a195700 1 -- 172.21.15.114:0/3996666228 >> v1:172.21.15.114:6789/0 conn(0x7fce541065a0 legacy=0x7fce54108990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce5a195700 1 -- 172.21.15.114:0/3996666228 shutdown_connections 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce5a195700 1 -- 172.21.15.114:0/3996666228 >> 172.21.15.114:0/3996666228 conn(0x7fce540fff70 msgr2=0x7fce541023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce5a195700 1 -- 172.21.15.114:0/3996666228 shutdown_connections 2024-03-24T11:34:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.950+0000 7fce5a195700 1 -- 172.21.15.114:0/3996666228 wait complete. 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.951+0000 7fce5a195700 1 Processor -- start 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.951+0000 7fce5a195700 1 -- start start 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.951+0000 7fce5a195700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce54199b20 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.951+0000 7fce537fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fce541065a0 0x7fce54199410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48358/0 (socket says 172.21.15.114:48358) 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.951+0000 7fce537fe700 1 -- 172.21.15.114:0/4054313714 learned_addr learned my addr 172.21.15.114:0/4054313714 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066862535 0 0) 0x7fce54199b20 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce400054f0 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3714659059 0 0) 0x7fce400054f0 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce54199b20 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fce3c004870 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2649608918 0 0) 0x7fce54199b20 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54199cb0 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce54199f60 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5419dab0 con 0x7fce541065a0 2024-03-24T11:34:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fce3c004a40 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.952+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fce3c006060 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.953+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 45723+0+0 (unknown 2752121932 0 0) 0x7fce3c011530 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.953+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce34005150 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.953+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7fce3c046e70 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:12.958+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fce3c012d50 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.204+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fce34005400 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.531+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (unknown 2263024820 0 0) 0x7fce3c014e30 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.532+0000 7fce50ff9700 1 -- 172.21.15.114:0/4054313714 <== mon.0 v1:172.21.15.114:6789/0 11 ==== mgrmap(e 5) v1 ==== 45734+0+0 (unknown 2184921475 0 0) 0x7fce3c045860 con 0x7fce541065a0 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.534+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 >> v1:172.21.15.114:6800/1606950246 conn(0x7fce40039ef0 legacy=0x7fce4003c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.534+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 >> v1:172.21.15.114:6789/0 conn(0x7fce541065a0 legacy=0x7fce54199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.534+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 shutdown_connections 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.534+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 >> 172.21.15.114:0/4054313714 conn(0x7fce540fff70 msgr2=0x7fce54190240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.534+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 shutdown_connections 2024-03-24T11:34:14.125 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:13.534+0000 7fce5a195700 1 -- 172.21.15.114:0/4054313714 wait complete. 2024-03-24T11:34:14.427 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:14.186+0000 7fda653d6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:34:14.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4054313714' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:34:14.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:14 smithi114 ceph-mon[30312]: mgrmap e5: y(active, since 6s) 2024-03-24T11:34:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:14.846+0000 7fda653d6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:34:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:14.938+0000 7fda653d6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:34:15.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.527+0000 7fcc84f93700 1 Processor -- start 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.527+0000 7fcc84f93700 1 -- start start 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.527+0000 7fcc84f93700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcc800796c0 con 0x7fcc80104420 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.527+0000 7fcc7e59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fcc80104420 0x7fcc80104840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48378/0 (socket says 172.21.15.114:48378) 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.527+0000 7fcc7e59c700 1 -- 172.21.15.114:0/2757787189 learned_addr learned my addr 172.21.15.114:0/2757787189 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.528+0000 7fcc7d59a700 1 -- 172.21.15.114:0/2757787189 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381529431 0 0) 0x7fcc800796c0 con 0x7fcc80104420 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.528+0000 7fcc7d59a700 1 -- 172.21.15.114:0/2757787189 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc680035b0 con 0x7fcc80104420 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.528+0000 7fcc7d59a700 1 -- 172.21.15.114:0/2757787189 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 630783273 0 0) 0x7fcc680035b0 con 0x7fcc80104420 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.528+0000 7fcc7d59a700 1 -- 172.21.15.114:0/2757787189 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc80105f60 con 0x7fcc80104420 2024-03-24T11:34:15.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.528+0000 7fcc7d59a700 1 -- 172.21.15.114:0/2757787189 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fcc70001580 con 0x7fcc80104420 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.528+0000 7fcc7d59a700 1 -- 172.21.15.114:0/2757787189 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fcc70001c10 con 0x7fcc80104420 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.528+0000 7fcc7d59a700 1 -- 172.21.15.114:0/2757787189 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fcc700057b0 con 0x7fcc80104420 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.529+0000 7fcc84f93700 1 -- 172.21.15.114:0/2757787189 >> v1:172.21.15.114:6789/0 conn(0x7fcc80104420 legacy=0x7fcc80104840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.529+0000 7fcc84f93700 1 -- 172.21.15.114:0/2757787189 shutdown_connections 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.529+0000 7fcc84f93700 1 -- 172.21.15.114:0/2757787189 >> 172.21.15.114:0/2757787189 conn(0x7fcc800fffa0 msgr2=0x7fcc80102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.529+0000 7fcc84f93700 1 -- 172.21.15.114:0/2757787189 shutdown_connections 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.529+0000 7fcc84f93700 1 -- 172.21.15.114:0/2757787189 wait complete. 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.529+0000 7fcc84f93700 1 Processor -- start 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc84f93700 1 -- start start 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc84f93700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcc8019bd10 con 0x7fcc80104420 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc7e59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fcc80104420 0x7fcc8019b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48390/0 (socket says 172.21.15.114:48390) 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc7e59c700 1 -- 172.21.15.114:0/2101246928 learned_addr learned my addr 172.21.15.114:0/2101246928 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570251184 0 0) 0x7fcc8019bd10 con 0x7fcc80104420 2024-03-24T11:34:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68005540 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641267009 0 0) 0x7fcc68005540 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc8019bd10 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fcc70001080 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.530+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3251652566 0 0) 0x7fcc8019bd10 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.531+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8019bea0 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.531+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8019c150 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.531+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8019fca0 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.531+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fcc700049d0 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.531+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fcc700061a0 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.532+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 5) v1 ==== 45734+0+0 (unknown 2184921475 0 0) 0x7fcc700115d0 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.532+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7fcc70047e60 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.532+0000 7fcc7dd9b700 1 -- 172.21.15.114:0/2101246928 >> v1:172.21.15.114:6800/1606950246 conn(0x7fcc68039f00 legacy=0x7fcc6803c3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/1606950246 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.532+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc60005150 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.538+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fcc70012e30 con 0x7fcc80104420 2024-03-24T11:34:15.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.733+0000 7fcc7dd9b700 1 -- 172.21.15.114:0/2101246928 >> v1:172.21.15.114:6800/1606950246 conn(0x7fcc68039f00 legacy=0x7fcc6803c3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/1606950246 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.793+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fcc60005c10 con 0x7fcc80104420 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.793+0000 7fcc6f7fe700 1 -- 172.21.15.114:0/2101246928 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7fcc70014f10 con 0x7fcc80104420 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.795+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 >> v1:172.21.15.114:6800/1606950246 conn(0x7fcc68039f00 legacy=0x7fcc6803c3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.795+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 >> v1:172.21.15.114:6789/0 conn(0x7fcc80104420 legacy=0x7fcc8019b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.795+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 shutdown_connections 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.795+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 >> 172.21.15.114:0/2101246928 conn(0x7fcc800fffa0 msgr2=0x7fcc80102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.796+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 shutdown_connections 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:14.796+0000 7fcc84f93700 1 -- 172.21.15.114:0/2101246928 wait complete. 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-03-24T11:34:15.300 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-03-24T11:34:15.415 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:15.081+0000 7fda653d6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:34:15.415 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:15.175+0000 7fda653d6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:34:15.415 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:15.258+0000 7fda653d6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:34:15.717 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2101246928' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:34:15.717 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:15.530+0000 7fda653d6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:34:15.717 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:15.641+0000 7fda653d6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:15.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:15.731+0000 7fda653d6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:34:15.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:15 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:15.825+0000 7fda653d6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:34:16.327 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:16.001+0000 7fda653d6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:34:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:16.943+0000 7fda653d6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:34:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:17.072+0000 7fda653d6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:34:17.688 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:17.403+0000 7fda653d6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:34:17.688 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:17.493+0000 7fda653d6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:34:17.688 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:17.686+0000 7fda653d6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:17.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:17.869+0000 7fda653d6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:34:18.405 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:18 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:18.148+0000 7fda653d6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:18.405 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:18 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:18.403+0000 7fda653d6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:18.730 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:18 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:18.638+0000 7fda653d6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:34:19.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:19 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:19.430+0000 7fda653d6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:34:20.310 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:19 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:19.997+0000 7fda653d6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:34:20.311 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:20.094+0000 7fda653d6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:34:20.311 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:20.309+0000 7fda653d6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: Active manager daemon y restarted 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: Activating manager daemon y 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: mgrmap e6: y(active, starting, since 0.00352514s) 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:34:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: Manager daemon y is now available 2024-03-24T11:34:20.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:20.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:20.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:20.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:20 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:21.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: Found migration_current of "None". Setting to last migration. 2024-03-24T11:34:21.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:21.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:34:21.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:34:21.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:21.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:21.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:21.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:21 smithi114 ceph-mon[30312]: mgrmap e7: y(active, since 1.00613s) 2024-03-24T11:34:21.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:21.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T11:34:21.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:34:21.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:21.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.719+0000 7fd73c88f700 1 Processor -- start 2024-03-24T11:34:21.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.719+0000 7fd73c88f700 1 -- start start 2024-03-24T11:34:21.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.719+0000 7fd73c88f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7380796e0 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.719+0000 7fd73659c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd738106610 0x7fd738108a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48402/0 (socket says 172.21.15.114:48402) 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.719+0000 7fd73659c700 1 -- 172.21.15.114:0/3858606364 learned_addr learned my addr 172.21.15.114:0/3858606364 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.719+0000 7fd73559a700 1 -- 172.21.15.114:0/3858606364 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847028155 0 0) 0x7fd7380796e0 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.719+0000 7fd73559a700 1 -- 172.21.15.114:0/3858606364 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7200035b0 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73559a700 1 -- 172.21.15.114:0/3858606364 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815313279 0 0) 0x7fd7200035b0 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73559a700 1 -- 172.21.15.114:0/3858606364 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73810b130 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73559a700 1 -- 172.21.15.114:0/3858606364 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fd72c001580 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73559a700 1 -- 172.21.15.114:0/3858606364 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fd72c001ce0 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73559a700 1 -- 172.21.15.114:0/3858606364 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fd72c005960 con 0x7fd738106610 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73c88f700 1 -- 172.21.15.114:0/3858606364 >> v1:172.21.15.114:6789/0 conn(0x7fd738106610 legacy=0x7fd738108a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73c88f700 1 -- 172.21.15.114:0/3858606364 shutdown_connections 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73c88f700 1 -- 172.21.15.114:0/3858606364 >> 172.21.15.114:0/3858606364 conn(0x7fd7380fffa0 msgr2=0x7fd738102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.720+0000 7fd73c88f700 1 -- 172.21.15.114:0/3858606364 shutdown_connections 2024-03-24T11:34:21.938 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.721+0000 7fd73c88f700 1 -- 172.21.15.114:0/3858606364 wait complete. 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.721+0000 7fd73c88f700 1 Processor -- start 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.721+0000 7fd73c88f700 1 -- start start 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.721+0000 7fd73c88f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd738199b30 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd73659c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd738106610 0x7fd738199420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48408/0 (socket says 172.21.15.114:48408) 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd73659c700 1 -- 172.21.15.114:0/3903666756 learned_addr learned my addr 172.21.15.114:0/3903666756 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557748379 0 0) 0x7fd738199b30 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7200054f0 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3127214088 0 0) 0x7fd7200054f0 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd738199b30 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fd72c004870 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 455829226 0 0) 0x7fd738199b30 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738199cc0 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd738199f70 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73819dac0 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fd72c001190 con 0x7fd738106610 2024-03-24T11:34:21.939 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.722+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fd72c006140 con 0x7fd738106610 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.723+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 5) v1 ==== 45734+0+0 (unknown 2184921475 0 0) 0x7fd72c011630 con 0x7fd738106610 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.724+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6800/1606950246 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd72003dbc0 con 0x7fd720039ef0 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.724+0000 7fd735d9b700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6800/1606950246 conn(0x7fd720039ef0 legacy=0x7fd72003c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/1606950246 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.724+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3423685221 0 0) 0x7fd72c047050 con 0x7fd738106610 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:15.925+0000 7fd735d9b700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6800/1606950246 conn(0x7fd720039ef0 legacy=0x7fd72003c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/1606950246 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:16.326+0000 7fd735d9b700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6800/1606950246 conn(0x7fd720039ef0 legacy=0x7fd72003c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/1606950246 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:17.127+0000 7fd735d9b700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6800/1606950246 conn(0x7fd720039ef0 legacy=0x7fd72003c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/1606950246 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:18.728+0000 7fd735d9b700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6800/1606950246 conn(0x7fd720039ef0 legacy=0x7fd72003c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/1606950246 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:20.324+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mgrmap(e 6) v1 ==== 45536+0+0 (unknown 1075322487 0 0) 0x7fd72c013420 con 0x7fd738106610 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:20.324+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6800/1606950246 conn(0x7fd720039ef0 legacy=0x7fd72003c3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.326+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mgrmap(e 7) v1 ==== 45628+0+0 (unknown 1131012136 0 0) 0x7fd72c048300 con 0x7fd738106610 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.326+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6800/773139778 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd72003dbc0 con 0x7fd72003e910 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.327+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (unknown 0 0 4109177037) 0x7fd72003dbc0 con 0x7fd72003e910 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 --> v1:172.21.15.114:6800/773139778 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd718002630 con 0x7fd72003e910 2024-03-24T11:34:21.940 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd7277fe700 1 -- 172.21.15.114:0/3903666756 <== mgr.14120 v1:172.21.15.114:6800/773139778 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (unknown 0 0 96372106) 0x7fd718002630 con 0x7fd72003e910 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6800/773139778 conn(0x7fd72003e910 legacy=0x7fd720040d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 >> v1:172.21.15.114:6789/0 conn(0x7fd738106610 legacy=0x7fd738199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 shutdown_connections 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 >> 172.21.15.114:0/3903666756 conn(0x7fd7380fffa0 msgr2=0x7fd738190270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 shutdown_connections 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:21.334+0000 7fd73c88f700 1 -- 172.21.15.114:0/3903666756 wait complete. 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-03-24T11:34:21.941 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-03-24T11:34:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:22 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:20] ENGINE Bus STARTING 2024-03-24T11:34:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:22 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:21] ENGINE Serving on http://172.21.15.114:8765 2024-03-24T11:34:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:22 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:21] ENGINE Serving on https://172.21.15.114:7150 2024-03-24T11:34:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:22 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:21] ENGINE Bus STARTED 2024-03-24T11:34:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:22 smithi114 ceph-mon[30312]: from='client.14124 v1:172.21.15.114:0/3903666756' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:34:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:22 smithi114 ceph-mon[30312]: from='client.14124 v1:172.21.15.114:0/3903666756' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:34:23.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.395+0000 7f6b5f34e700 1 Processor -- start 2024-03-24T11:34:23.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.395+0000 7f6b5f34e700 1 -- start start 2024-03-24T11:34:23.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.395+0000 7f6b5f34e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6b58108e20 con 0x7f6b581049f0 2024-03-24T11:34:23.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.395+0000 7f6b5d0ea700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6b581049f0 0x7f6b58104e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46416/0 (socket says 172.21.15.114:46416) 2024-03-24T11:34:23.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.395+0000 7f6b5d0ea700 1 -- 172.21.15.114:0/3905337833 learned_addr learned my addr 172.21.15.114:0/3905337833 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:23.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.395+0000 7f6b4ffff700 1 -- 172.21.15.114:0/3905337833 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450616368 0 0) 0x7f6b58108e20 con 0x7f6b581049f0 2024-03-24T11:34:23.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.395+0000 7f6b4ffff700 1 -- 172.21.15.114:0/3905337833 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b440035b0 con 0x7f6b581049f0 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b4ffff700 1 -- 172.21.15.114:0/3905337833 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4125127984 0 0) 0x7f6b440035b0 con 0x7f6b581049f0 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b4ffff700 1 -- 172.21.15.114:0/3905337833 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b58109fc0 con 0x7f6b581049f0 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b4ffff700 1 -- 172.21.15.114:0/3905337833 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6b54001580 con 0x7f6b581049f0 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b4ffff700 1 -- 172.21.15.114:0/3905337833 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f6b54001ce0 con 0x7f6b581049f0 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b4ffff700 1 -- 172.21.15.114:0/3905337833 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6b54005960 con 0x7f6b581049f0 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b5f34e700 1 -- 172.21.15.114:0/3905337833 >> v1:172.21.15.114:6789/0 conn(0x7f6b581049f0 legacy=0x7f6b58104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b5f34e700 1 -- 172.21.15.114:0/3905337833 shutdown_connections 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b5f34e700 1 -- 172.21.15.114:0/3905337833 >> 172.21.15.114:0/3905337833 conn(0x7f6b580fff70 msgr2=0x7f6b581023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b5f34e700 1 -- 172.21.15.114:0/3905337833 shutdown_connections 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.396+0000 7f6b5f34e700 1 -- 172.21.15.114:0/3905337833 wait complete. 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.397+0000 7f6b5f34e700 1 Processor -- start 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.397+0000 7f6b5f34e700 1 -- start start 2024-03-24T11:34:23.210 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.397+0000 7f6b5f34e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6b58197920 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.397+0000 7f6b5d0ea700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6b581049f0 0x7f6b58197210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46432/0 (socket says 172.21.15.114:46432) 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.397+0000 7f6b5d0ea700 1 -- 172.21.15.114:0/2733229842 learned_addr learned my addr 172.21.15.114:0/2733229842 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.397+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104676444 0 0) 0x7f6b58197920 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.397+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b440054f0 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 39606545 0 0) 0x7f6b440054f0 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b58197920 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6b54005940 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2298606218 0 0) 0x7f6b58197920 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b58197ab0 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b58197d60 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5819b8b0 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f6b54001b20 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.398+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6b540060b0 con 0x7f6b581049f0 2024-03-24T11:34:23.211 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.399+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7f6b540115a0 con 0x7f6b581049f0 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.399+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c005150 con 0x7f6b581049f0 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.399+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f6b54046fd0 con 0x7f6b581049f0 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.404+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f6b54012e10 con 0x7f6b581049f0 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.612+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 --> v1:172.21.15.114:6800/773139778 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b3c000bc0 con 0x7f6b4403c100 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.618+0000 7f6b4e7fc700 1 -- 172.21.15.114:0/2733229842 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f6b3c000bc0 con 0x7f6b4403c100 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.620+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 >> v1:172.21.15.114:6800/773139778 conn(0x7f6b4403c100 legacy=0x7f6b4403e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.620+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 >> v1:172.21.15.114:6789/0 conn(0x7f6b581049f0 legacy=0x7f6b58197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.620+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 shutdown_connections 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.620+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 >> 172.21.15.114:0/2733229842 conn(0x7f6b580fff70 msgr2=0x7f6b581023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.620+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 shutdown_connections 2024-03-24T11:34:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:22.620+0000 7f6b5f34e700 1 -- 172.21.15.114:0/2733229842 wait complete. 2024-03-24T11:34:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:23 smithi114 ceph-mon[30312]: mgrmap e8: y(active, since 2s) 2024-03-24T11:34:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:23 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:23 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:24.548 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T11:34:24.548 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.862+0000 7f578c0b8700 1 Processor -- start 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.862+0000 7f578c0b8700 1 -- start start 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.862+0000 7f578c0b8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5784079720 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5789e54700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5784104390 0x7f57841047b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46440/0 (socket says 172.21.15.114:46440) 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5789e54700 1 -- 172.21.15.114:0/3660384520 learned_addr learned my addr 172.21.15.114:0/3660384520 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5788e52700 1 -- 172.21.15.114:0/3660384520 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431482737 0 0) 0x7f5784079720 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5788e52700 1 -- 172.21.15.114:0/3660384520 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57700035b0 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5788e52700 1 -- 172.21.15.114:0/3660384520 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619777253 0 0) 0x7f57700035b0 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5788e52700 1 -- 172.21.15.114:0/3660384520 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5784105ed0 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5788e52700 1 -- 172.21.15.114:0/3660384520 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5780001580 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5788e52700 1 -- 172.21.15.114:0/3660384520 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f5780001ce0 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.863+0000 7f5788e52700 1 -- 172.21.15.114:0/3660384520 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5780005960 con 0x7f5784104390 2024-03-24T11:34:24.549 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.864+0000 7f578c0b8700 1 -- 172.21.15.114:0/3660384520 >> v1:172.21.15.114:6789/0 conn(0x7f5784104390 legacy=0x7f57841047b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.864+0000 7f578c0b8700 1 -- 172.21.15.114:0/3660384520 shutdown_connections 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.864+0000 7f578c0b8700 1 -- 172.21.15.114:0/3660384520 >> 172.21.15.114:0/3660384520 conn(0x7f57840fff70 msgr2=0x7f57841023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.864+0000 7f578c0b8700 1 -- 172.21.15.114:0/3660384520 shutdown_connections 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.864+0000 7f578c0b8700 1 -- 172.21.15.114:0/3660384520 wait complete. 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.864+0000 7f578c0b8700 1 Processor -- start 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f578c0b8700 1 -- start start 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f578c0b8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f578419bd40 con 0x7f5784104390 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f5789e54700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5784104390 0x7f578419b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46448/0 (socket says 172.21.15.114:46448) 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f5789e54700 1 -- 172.21.15.114:0/2798331809 learned_addr learned my addr 172.21.15.114:0/2798331809 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706488138 0 0) 0x7f578419bd40 con 0x7f5784104390 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57700054f0 con 0x7f5784104390 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338742348 0 0) 0x7f57700054f0 con 0x7f5784104390 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f578419bd40 con 0x7f5784104390 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5780004870 con 0x7f5784104390 2024-03-24T11:34:24.550 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2420580038 0 0) 0x7f578419bd40 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578419bed0 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578419c180 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.865+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f578419fcd0 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.866+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f5780004a40 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.866+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5780006070 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.866+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7f5780011540 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.866+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5768005120 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.866+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f5780046fb0 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:23.872+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f5780012db0 con 0x7f5784104390 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.063+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 --> v1:172.21.15.114:6800/773139778 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f5768000b90 con 0x7f5770039f30 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.064+0000 7f577affd700 1 -- 172.21.15.114:0/2798331809 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (unknown 0 0 2070689548) 0x7f5768000b90 con 0x7f5770039f30 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.066+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 >> v1:172.21.15.114:6800/773139778 conn(0x7f5770039f30 legacy=0x7f577003c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.066+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 >> v1:172.21.15.114:6789/0 conn(0x7f5784104390 legacy=0x7f578419b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:24.551 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.066+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 shutdown_connections 2024-03-24T11:34:24.552 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.066+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 >> 172.21.15.114:0/2798331809 conn(0x7f57840fff70 msgr2=0x7f57841023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:24.552 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.066+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 shutdown_connections 2024-03-24T11:34:24.552 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:24.066+0000 7f578c0b8700 1 -- 172.21.15.114:0/2798331809 wait complete. 2024-03-24T11:34:24.552 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-03-24T11:34:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:24 smithi114 ceph-mon[30312]: from='client.14132 v1:172.21.15.114:0/2733229842' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:25.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-mon[30312]: from='client.14134 v1:172.21.15.114:0/2798331809' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:25.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: Generating public/private rsa key pair. 2024-03-24T11:34:25.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: Your identification has been saved in /tmp/tmpafqx3kdu/key. 2024-03-24T11:34:25.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: Your public key has been saved in /tmp/tmpafqx3kdu/key.pub. 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: The key fingerprint is: 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: SHA256:kgTMlBBu0geP2dHRGIS9nGRfP+qbWDphsx24sGhO+AE ceph-412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: The key's randomart image is: 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: +---[RSA 3072]----+ 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | +*+*+= | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | o *=o* . . | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: |. * ++.+ . . | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | o . .+.. o | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | E o S. . . | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | o ..= o | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | . o. + B.. | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | oo.. ++o. | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: | oo o.o. | 2024-03-24T11:34:25.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: +----[SHA256]-----+ 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.010+0000 7f6ee2459700 1 Processor -- start 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.010+0000 7f6ee2459700 1 -- start start 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.010+0000 7f6ee2459700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6edc0796e0 con 0x7f6edc106600 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edbfff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6edc106600 0x7f6edc1089f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46456/0 (socket says 172.21.15.114:46456) 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edbfff700 1 -- 172.21.15.114:0/1811007051 learned_addr learned my addr 172.21.15.114:0/1811007051 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edaffd700 1 -- 172.21.15.114:0/1811007051 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285198286 0 0) 0x7f6edc0796e0 con 0x7f6edc106600 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edaffd700 1 -- 172.21.15.114:0/1811007051 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec40035b0 con 0x7f6edc106600 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edaffd700 1 -- 172.21.15.114:0/1811007051 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3832306886 0 0) 0x7f6ec40035b0 con 0x7f6edc106600 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edaffd700 1 -- 172.21.15.114:0/1811007051 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc10b120 con 0x7f6edc106600 2024-03-24T11:34:26.030 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edaffd700 1 -- 172.21.15.114:0/1811007051 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6ecc001580 con 0x7f6edc106600 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edaffd700 1 -- 172.21.15.114:0/1811007051 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f6ecc001c10 con 0x7f6edc106600 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.011+0000 7f6edaffd700 1 -- 172.21.15.114:0/1811007051 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6ecc0057b0 con 0x7f6edc106600 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.012+0000 7f6ee2459700 1 -- 172.21.15.114:0/1811007051 >> v1:172.21.15.114:6789/0 conn(0x7f6edc106600 legacy=0x7f6edc1089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.012+0000 7f6ee2459700 1 -- 172.21.15.114:0/1811007051 shutdown_connections 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.012+0000 7f6ee2459700 1 -- 172.21.15.114:0/1811007051 >> 172.21.15.114:0/1811007051 conn(0x7f6edc0fffd0 msgr2=0x7f6edc1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.012+0000 7f6ee2459700 1 -- 172.21.15.114:0/1811007051 shutdown_connections 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.012+0000 7f6ee2459700 1 -- 172.21.15.114:0/1811007051 wait complete. 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.012+0000 7f6ee2459700 1 Processor -- start 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.012+0000 7f6ee2459700 1 -- start start 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ee2459700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6edc199b50 con 0x7f6edc106600 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6edbfff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6edc106600 0x7f6edc199440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46468/0 (socket says 172.21.15.114:46468) 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6edbfff700 1 -- 172.21.15.114:0/1625905684 learned_addr learned my addr 172.21.15.114:0/1625905684 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764599770 0 0) 0x7f6edc199b50 con 0x7f6edc106600 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec4005540 con 0x7f6edc106600 2024-03-24T11:34:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 187361312 0 0) 0x7f6ec4005540 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6edc199b50 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6ecc001080 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2605046225 0 0) 0x7f6edc199b50 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.013+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc199ce0 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.014+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc199f90 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.014+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc19dae0 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.014+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f6ecc004900 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.014+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6ecc006250 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.014+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7f6ecc011680 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.014+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec8005120 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.015+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f6ecc047190 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.019+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f6ecc012ee0 con 0x7f6edc106600 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.212+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 --> v1:172.21.15.114:6800/773139778 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ec8000b90 con 0x7f6ec4039e90 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.488+0000 7f6ed97fa700 1 -- 172.21.15.114:0/1625905684 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f6ec8000b90 con 0x7f6ec4039e90 2024-03-24T11:34:26.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.490+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 >> v1:172.21.15.114:6800/773139778 conn(0x7f6ec4039e90 legacy=0x7f6ec403c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:26.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.490+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 >> v1:172.21.15.114:6789/0 conn(0x7f6edc106600 legacy=0x7f6edc199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:26.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.491+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 shutdown_connections 2024-03-24T11:34:26.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.491+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 >> 172.21.15.114:0/1625905684 conn(0x7f6edc0fffd0 msgr2=0x7f6edc190260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:26.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.491+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 shutdown_connections 2024-03-24T11:34:26.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:25.491+0000 7f6ee2459700 1 -- 172.21.15.114:0/1625905684 wait complete. 2024-03-24T11:34:26.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:26 smithi114 ceph-mon[30312]: from='client.14136 v1:172.21.15.114:0/1625905684' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:26.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:26 smithi114 ceph-mon[30312]: Generating ssh key... 2024-03-24T11:34:26.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:26 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:26.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:26 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChYOvg9D3XtfRdVTk6lCNk6Pd8GmaZkIA+S7+s4vTfm1II6DTu5T8/bL9uZuA4p/arWEys7qlCcFK/wNDUTMihuj81pNSW8y3696Uox0IJsfEG17xk9PhLbPdmNLtovNfvCcHobUW4d5ryaPGiBFC0vEKKpTYFCM+7vBoSZxB1OQz2GEG9Wo6EJc09KDx3v2vwH8ExdzVh1sIH97vCKZ78hwe6oi4cEQ9B3n+adhUz97QA/7X+RrxwogZukK4Ub5SB6oxl2bIk/EATMnXEDbqPfwGi8aHldlcpEkFPqUi+TKP0d4GKoYBbjIzOnp4Cq/kH2jnAOx9wWfB0wDCTFKpLeREfUEHYOZuurJztFqu6OhdjUiQJcltw52i6+tOheVULaIhHDBRJ4Rlf/LeHRyDModvI8mjT5xu+lf35Lz4gz+GW5nbjCFX2Fw4k+om1KBcA1I0rkcJFpo3MDg4s0hEvoj9F5p+cL9gEn2tHZOT0AKF3/TuacTchbxEt1s4Rs3M= ceph-412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.525+0000 7ff7cf2bc700 1 Processor -- start 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.525+0000 7ff7cf2bc700 1 -- start start 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7cf2bc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7c8079720 con 0x7ff7c81043b0 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7cd058700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff7c81043b0 0x7ff7c81047d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46478/0 (socket says 172.21.15.114:46478) 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7cd058700 1 -- 172.21.15.114:0/499207516 learned_addr learned my addr 172.21.15.114:0/499207516 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7bffff700 1 -- 172.21.15.114:0/499207516 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456054748 0 0) 0x7ff7c8079720 con 0x7ff7c81043b0 2024-03-24T11:34:27.245 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7bffff700 1 -- 172.21.15.114:0/499207516 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b40035b0 con 0x7ff7c81043b0 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7bffff700 1 -- 172.21.15.114:0/499207516 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 289705608 0 0) 0x7ff7b40035b0 con 0x7ff7c81043b0 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7bffff700 1 -- 172.21.15.114:0/499207516 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8105ef0 con 0x7ff7c81043b0 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7bffff700 1 -- 172.21.15.114:0/499207516 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff7b8001580 con 0x7ff7c81043b0 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7bffff700 1 -- 172.21.15.114:0/499207516 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7ff7b8001c10 con 0x7ff7c81043b0 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.526+0000 7ff7bffff700 1 -- 172.21.15.114:0/499207516 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff7b80057b0 con 0x7ff7c81043b0 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.527+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/499207516 >> v1:172.21.15.114:6789/0 conn(0x7ff7c81043b0 legacy=0x7ff7c81047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.527+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/499207516 shutdown_connections 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.527+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/499207516 >> 172.21.15.114:0/499207516 conn(0x7ff7c80fffd0 msgr2=0x7ff7c81023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.527+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/499207516 shutdown_connections 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.527+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/499207516 wait complete. 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7cf2bc700 1 Processor -- start 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7cf2bc700 1 -- start start 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7cf2bc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7c819bd40 con 0x7ff7c81043b0 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7cd058700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff7c81043b0 0x7ff7c819b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46490/0 (socket says 172.21.15.114:46490) 2024-03-24T11:34:27.246 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7cd058700 1 -- 172.21.15.114:0/1364258515 learned_addr learned my addr 172.21.15.114:0/1364258515 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811075563 0 0) 0x7ff7c819bd40 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b4005540 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211084638 0 0) 0x7ff7b4005540 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7c819bd40 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.528+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff7b8001080 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3524988473 0 0) 0x7ff7c819bd40 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c819bed0 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c819c180 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c819fcd0 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7ff7b80049d0 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff7b80061a0 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7ff7b8011690 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.529+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ac005150 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.530+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7ff7b8047480 con 0x7ff7c81043b0 2024-03-24T11:34:27.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.534+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ff7b8013480 con 0x7ff7c81043b0 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.728+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 --> v1:172.21.15.114:6800/773139778 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7ac000bc0 con 0x7ff7b4039e90 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.729+0000 7ff7be7fc700 1 -- 172.21.15.114:0/1364258515 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (unknown 0 0 3482335596) 0x7ff7ac000bc0 con 0x7ff7b4039e90 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.730+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 >> v1:172.21.15.114:6800/773139778 conn(0x7ff7b4039e90 legacy=0x7ff7b403c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.730+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 >> v1:172.21.15.114:6789/0 conn(0x7ff7c81043b0 legacy=0x7ff7c819b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.731+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 shutdown_connections 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.731+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 >> 172.21.15.114:0/1364258515 conn(0x7ff7c80fffd0 msgr2=0x7ff7c81023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.731+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 shutdown_connections 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:26.731+0000 7ff7cf2bc700 1 -- 172.21.15.114:0/1364258515 wait complete. 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T11:34:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-03-24T11:34:27.903 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:27 smithi114 ceph-mon[30312]: from='client.14138 v1:172.21.15.114:0/1364258515' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:28.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:28 smithi114 ceph-mon[30312]: from='client.14140 v1:172.21.15.114:0/305688865' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:28.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:28 smithi114 ceph-mon[30312]: Deploying cephadm binary to smithi114 2024-03-24T11:34:32.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-03-24T11:34:32.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.697+0000 7fa53df66700 1 Processor -- start 2024-03-24T11:34:32.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.697+0000 7fa53df66700 1 -- start start 2024-03-24T11:34:32.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.697+0000 7fa53df66700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa53807a370 con 0x7fa53807b770 2024-03-24T11:34:32.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5377fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa53807b770 0x7fa538079bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46500/0 (socket says 172.21.15.114:46500) 2024-03-24T11:34:32.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5377fe700 1 -- 172.21.15.114:0/3577980258 learned_addr learned my addr 172.21.15.114:0/3577980258 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5367fc700 1 -- 172.21.15.114:0/3577980258 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366769895 0 0) 0x7fa53807a370 con 0x7fa53807b770 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5367fc700 1 -- 172.21.15.114:0/3577980258 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5200035b0 con 0x7fa53807b770 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5367fc700 1 -- 172.21.15.114:0/3577980258 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2657364441 0 0) 0x7fa5200035b0 con 0x7fa53807b770 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5367fc700 1 -- 172.21.15.114:0/3577980258 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53807a500 con 0x7fa53807b770 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5367fc700 1 -- 172.21.15.114:0/3577980258 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa52c001580 con 0x7fa53807b770 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5367fc700 1 -- 172.21.15.114:0/3577980258 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fa52c001ce0 con 0x7fa53807b770 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.698+0000 7fa5367fc700 1 -- 172.21.15.114:0/3577980258 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa52c005960 con 0x7fa53807b770 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.699+0000 7fa53df66700 1 -- 172.21.15.114:0/3577980258 >> v1:172.21.15.114:6789/0 conn(0x7fa53807b770 legacy=0x7fa538079bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.699+0000 7fa53df66700 1 -- 172.21.15.114:0/3577980258 shutdown_connections 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.699+0000 7fa53df66700 1 -- 172.21.15.114:0/3577980258 >> 172.21.15.114:0/3577980258 conn(0x7fa5381010d0 msgr2=0x7fa5381034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.699+0000 7fa53df66700 1 -- 172.21.15.114:0/3577980258 shutdown_connections 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.699+0000 7fa53df66700 1 -- 172.21.15.114:0/3577980258 wait complete. 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.699+0000 7fa53df66700 1 Processor -- start 2024-03-24T11:34:32.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa53df66700 1 -- start start 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa53df66700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa53819bd80 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa5377fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa53807b770 0x7fa53819b670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46512/0 (socket says 172.21.15.114:46512) 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa5377fe700 1 -- 172.21.15.114:0/305688865 learned_addr learned my addr 172.21.15.114:0/305688865 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495294732 0 0) 0x7fa53819bd80 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5200054f0 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2052719911 0 0) 0x7fa5200054f0 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa53819bd80 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa52c004870 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 405664066 0 0) 0x7fa53819bd80 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53819bf10 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53819c1c0 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.700+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53819fd10 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.701+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fa52c004a40 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.701+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fa52c006260 con 0x7fa53807b770 2024-03-24T11:34:32.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.701+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7fa52c011670 con 0x7fa53807b770 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.701+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53810cf80 con 0x7fa53807b770 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.701+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7fa52c046fd0 con 0x7fa53807b770 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.706+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fa52c012da0 con 0x7fa53807b770 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:27.903+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 --> v1:172.21.15.114:6800/773139778 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}) v1 -- 0x7fa53810bd60 con 0x7fa520039f30 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:32.166+0000 7fa534ff9700 1 -- 172.21.15.114:0/305688865 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 37019767) 0x7fa53810bd60 con 0x7fa520039f30 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:32.168+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 >> v1:172.21.15.114:6800/773139778 conn(0x7fa520039f30 legacy=0x7fa52003c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:32.168+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 >> v1:172.21.15.114:6789/0 conn(0x7fa53807b770 legacy=0x7fa53819b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:32.168+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 shutdown_connections 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:32.168+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 >> 172.21.15.114:0/305688865 conn(0x7fa5381010d0 msgr2=0x7fa53810a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:32.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:32.168+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 shutdown_connections 2024-03-24T11:34:32.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:32.169+0000 7fa53df66700 1 -- 172.21.15.114:0/305688865 wait complete. 2024-03-24T11:34:32.782 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mon service... 2024-03-24T11:34:33.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:33 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:33.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:33 smithi114 ceph-mon[30312]: Added host smithi114 2024-03-24T11:34:33.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:33 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:34.232 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T11:34:34.232 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.213+0000 7f5faf693700 1 Processor -- start 2024-03-24T11:34:34.232 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.213+0000 7f5faf693700 1 -- start start 2024-03-24T11:34:34.232 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.213+0000 7f5faf693700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fa807a330 con 0x7f5fa807b730 2024-03-24T11:34:34.232 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.213+0000 7f5fad42f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5fa807b730 0x7f5fa8079b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43756/0 (socket says 172.21.15.114:43756) 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.213+0000 7f5fad42f700 1 -- 172.21.15.114:0/1562835522 learned_addr learned my addr 172.21.15.114:0/1562835522 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.213+0000 7f5f9ffff700 1 -- 172.21.15.114:0/1562835522 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672525346 0 0) 0x7f5fa807a330 con 0x7f5fa807b730 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.213+0000 7f5f9ffff700 1 -- 172.21.15.114:0/1562835522 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f940035b0 con 0x7f5fa807b730 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5f9ffff700 1 -- 172.21.15.114:0/1562835522 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021960639 0 0) 0x7f5f940035b0 con 0x7f5fa807b730 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5f9ffff700 1 -- 172.21.15.114:0/1562835522 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa807a4c0 con 0x7f5fa807b730 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5f9ffff700 1 -- 172.21.15.114:0/1562835522 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5fa4001580 con 0x7f5fa807b730 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5f9ffff700 1 -- 172.21.15.114:0/1562835522 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f5fa4001ce0 con 0x7f5fa807b730 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5f9ffff700 1 -- 172.21.15.114:0/1562835522 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5fa4005960 con 0x7f5fa807b730 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5faf693700 1 -- 172.21.15.114:0/1562835522 >> v1:172.21.15.114:6789/0 conn(0x7f5fa807b730 legacy=0x7f5fa8079b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5faf693700 1 -- 172.21.15.114:0/1562835522 shutdown_connections 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.214+0000 7f5faf693700 1 -- 172.21.15.114:0/1562835522 >> 172.21.15.114:0/1562835522 conn(0x7f5fa81010b0 msgr2=0x7f5fa81034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.215+0000 7f5faf693700 1 -- 172.21.15.114:0/1562835522 shutdown_connections 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.215+0000 7f5faf693700 1 -- 172.21.15.114:0/1562835522 wait complete. 2024-03-24T11:34:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.215+0000 7f5faf693700 1 Processor -- start 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.215+0000 7f5faf693700 1 -- start start 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.215+0000 7f5faf693700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fa819bd60 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.215+0000 7f5fad42f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5fa807b730 0x7f5fa819b650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43762/0 (socket says 172.21.15.114:43762) 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.215+0000 7f5fad42f700 1 -- 172.21.15.114:0/3923440603 learned_addr learned my addr 172.21.15.114:0/3923440603 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409382258 0 0) 0x7f5fa819bd60 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f940054f0 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879494479 0 0) 0x7f5f940054f0 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fa819bd60 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5fa4004870 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3277523958 0 0) 0x7f5fa819bd60 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa819bef0 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa819c1a0 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa819fcf0 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7f5fa4001190 con 0x7f5fa807b730 2024-03-24T11:34:34.234 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.216+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f5fa4006130 con 0x7f5fa807b730 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.217+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7f5fa4011560 con 0x7f5fa807b730 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.217+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c005150 con 0x7f5fa807b730 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.217+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f5fa4047050 con 0x7f5fa807b730 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.222+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f5fa4012dc0 con 0x7f5fa807b730 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.427+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 --> v1:172.21.15.114:6800/773139778 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5f8c000bc0 con 0x7f5f94039f40 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.434+0000 7f5f9e7fc700 1 -- 172.21.15.114:0/3923440603 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f5f8c000bc0 con 0x7f5f94039f40 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.436+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 >> v1:172.21.15.114:6800/773139778 conn(0x7f5f94039f40 legacy=0x7f5f9403c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.436+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 >> v1:172.21.15.114:6789/0 conn(0x7f5fa807b730 legacy=0x7f5fa819b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.436+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 shutdown_connections 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.436+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 >> 172.21.15.114:0/3923440603 conn(0x7f5fa81010b0 msgr2=0x7f5fa8109fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.437+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 shutdown_connections 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:33.437+0000 7f5faf693700 1 -- 172.21.15.114:0/3923440603 wait complete. 2024-03-24T11:34:34.235 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mgr service... 2024-03-24T11:34:34.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:34 smithi114 ceph-mon[30312]: from='client.14142 v1:172.21.15.114:0/3923440603' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:34 smithi114 ceph-mon[30312]: Saving service mon spec with placement count:5 2024-03-24T11:34:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:34 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.660+0000 7fab4f248700 1 Processor -- start 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.660+0000 7fab4f248700 1 -- start start 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab4f248700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab48109f40 con 0x7fab48105b10 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab4cfe4700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fab48105b10 0x7fab48105f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43768/0 (socket says 172.21.15.114:43768) 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab4cfe4700 1 -- 172.21.15.114:0/2438671706 learned_addr learned my addr 172.21.15.114:0/2438671706 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab477fe700 1 -- 172.21.15.114:0/2438671706 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609608284 0 0) 0x7fab48109f40 con 0x7fab48105b10 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab477fe700 1 -- 172.21.15.114:0/2438671706 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab380035b0 con 0x7fab48105b10 2024-03-24T11:34:36.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab477fe700 1 -- 172.21.15.114:0/2438671706 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1609905084 0 0) 0x7fab380035b0 con 0x7fab48105b10 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab477fe700 1 -- 172.21.15.114:0/2438671706 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4810b0e0 con 0x7fab48105b10 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.661+0000 7fab477fe700 1 -- 172.21.15.114:0/2438671706 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fab3c001580 con 0x7fab48105b10 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.662+0000 7fab477fe700 1 -- 172.21.15.114:0/2438671706 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fab3c001ce0 con 0x7fab48105b10 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.662+0000 7fab477fe700 1 -- 172.21.15.114:0/2438671706 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fab3c005960 con 0x7fab48105b10 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.662+0000 7fab4f248700 1 -- 172.21.15.114:0/2438671706 >> v1:172.21.15.114:6789/0 conn(0x7fab48105b10 legacy=0x7fab48105f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.662+0000 7fab4f248700 1 -- 172.21.15.114:0/2438671706 shutdown_connections 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.662+0000 7fab4f248700 1 -- 172.21.15.114:0/2438671706 >> 172.21.15.114:0/2438671706 conn(0x7fab481010b0 msgr2=0x7fab481034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.662+0000 7fab4f248700 1 -- 172.21.15.114:0/2438671706 shutdown_connections 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.662+0000 7fab4f248700 1 -- 172.21.15.114:0/2438671706 wait complete. 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab4f248700 1 Processor -- start 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab4f248700 1 -- start start 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab4f248700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab48199b50 con 0x7fab48105b10 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab4cfe4700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fab48105b10 0x7fab48199440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43772/0 (socket says 172.21.15.114:43772) 2024-03-24T11:34:36.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab4cfe4700 1 -- 172.21.15.114:0/1730789177 learned_addr learned my addr 172.21.15.114:0/1730789177 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482051179 0 0) 0x7fab48199b50 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab380054f0 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.663+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4052950593 0 0) 0x7fab380054f0 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab48199b50 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fab3c005940 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2441969899 0 0) 0x7fab48199b50 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab48199ce0 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab48199f90 con 0x7fab48105b10 2024-03-24T11:34:36.019 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 3355376714 0 0) 0x7fab3c001b60 con 0x7fab48105b10 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fab3c006310 con 0x7fab48105b10 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.664+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4819dae0 con 0x7fab48105b10 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.665+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7fab3c011870 con 0x7fab48105b10 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.665+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4810abd0 con 0x7fab48105b10 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.665+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7fab3c0471b0 con 0x7fab48105b10 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.670+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fab3c0134f0 con 0x7fab48105b10 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.886+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 --> v1:172.21.15.114:6800/773139778 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fab480620b0 con 0x7fab38039ea0 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.890+0000 7fab45ffb700 1 -- 172.21.15.114:0/1730789177 <== mgr.14120 v1:172.21.15.114:6800/773139778 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7fab480620b0 con 0x7fab38039ea0 2024-03-24T11:34:36.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.892+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 >> v1:172.21.15.114:6800/773139778 conn(0x7fab38039ea0 legacy=0x7fab3803c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:36.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.892+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 >> v1:172.21.15.114:6789/0 conn(0x7fab48105b10 legacy=0x7fab48199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:36.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.892+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 shutdown_connections 2024-03-24T11:34:36.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.892+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 >> 172.21.15.114:0/1730789177 conn(0x7fab481010b0 msgr2=0x7fab481034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:36.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.893+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 shutdown_connections 2024-03-24T11:34:36.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:34.893+0000 7fab4f248700 1 -- 172.21.15.114:0/1730789177 wait complete. 2024-03-24T11:34:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:35 smithi114 ceph-mon[30312]: from='client.14144 v1:172.21.15.114:0/1730789177' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:35 smithi114 ceph-mon[30312]: Saving service mgr spec with placement count:2 2024-03-24T11:34:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:35 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:35 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:37.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:37 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:37.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2180337388' entity='client.admin' 2024-03-24T11:34:37.458 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.477+0000 7f205f574700 1 Processor -- start 2024-03-24T11:34:37.458 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.477+0000 7f205f574700 1 -- start start 2024-03-24T11:34:37.458 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.477+0000 7f205f574700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2058108e20 con 0x7f20581049f0 2024-03-24T11:34:37.458 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.477+0000 7f205d310700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f20581049f0 0x7f2058104e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43774/0 (socket says 172.21.15.114:43774) 2024-03-24T11:34:37.458 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.477+0000 7f205d310700 1 -- 172.21.15.114:0/1076467081 learned_addr learned my addr 172.21.15.114:0/1076467081 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:37.458 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.478+0000 7f204ffff700 1 -- 172.21.15.114:0/1076467081 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612687639 0 0) 0x7f2058108e20 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.478+0000 7f204ffff700 1 -- 172.21.15.114:0/1076467081 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20440035b0 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.478+0000 7f204ffff700 1 -- 172.21.15.114:0/1076467081 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3963848582 0 0) 0x7f20440035b0 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.478+0000 7f204ffff700 1 -- 172.21.15.114:0/1076467081 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2058109fc0 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.478+0000 7f204ffff700 1 -- 172.21.15.114:0/1076467081 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f2054001580 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.478+0000 7f204ffff700 1 -- 172.21.15.114:0/1076467081 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2054001c30 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.478+0000 7f204ffff700 1 -- 172.21.15.114:0/1076467081 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f2054005770 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.479+0000 7f205f574700 1 -- 172.21.15.114:0/1076467081 >> v1:172.21.15.114:6789/0 conn(0x7f20581049f0 legacy=0x7f2058104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.479+0000 7f205f574700 1 -- 172.21.15.114:0/1076467081 shutdown_connections 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.479+0000 7f205f574700 1 -- 172.21.15.114:0/1076467081 >> 172.21.15.114:0/1076467081 conn(0x7f20580fff70 msgr2=0x7f20581023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.479+0000 7f205f574700 1 -- 172.21.15.114:0/1076467081 shutdown_connections 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.479+0000 7f205f574700 1 -- 172.21.15.114:0/1076467081 wait complete. 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.479+0000 7f205f574700 1 Processor -- start 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.479+0000 7f205f574700 1 -- start start 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f205f574700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f205819f4b0 con 0x7f20581049f0 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f205d310700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f20581049f0 0x7f2058101e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43788/0 (socket says 172.21.15.114:43788) 2024-03-24T11:34:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f205d310700 1 -- 172.21.15.114:0/2180337388 learned_addr learned my addr 172.21.15.114:0/2180337388 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556055619 0 0) 0x7f205819f4b0 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2044005540 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540033569 0 0) 0x7f2044005540 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f205819f4b0 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f2054001080 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1929115536 0 0) 0x7f205819f4b0 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20581a0650 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205819f6a0 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f205819fba0 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.480+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2054004870 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.481+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f2054006100 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.481+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7f2054011530 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.481+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c005150 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.482+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f2054047050 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.490+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f2054012de0 con 0x7f20581049f0 2024-03-24T11:34:37.460 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.701+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 --> v1:172.21.15.114:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f203c005400 con 0x7f20581049f0 2024-03-24T11:34:37.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.703+0000 7f204e7fc700 1 -- 172.21.15.114:0/2180337388 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (unknown 2684808024 0 0) 0x7f2054014ec0 con 0x7f20581049f0 2024-03-24T11:34:37.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.705+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 >> v1:172.21.15.114:6800/773139778 conn(0x7f2044039f30 legacy=0x7f204403c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:37.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.705+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 >> v1:172.21.15.114:6789/0 conn(0x7f20581049f0 legacy=0x7f2058101e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:37.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.705+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 shutdown_connections 2024-03-24T11:34:37.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.705+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 >> 172.21.15.114:0/2180337388 conn(0x7f20580fff70 msgr2=0x7f2058078880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:37.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.706+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 shutdown_connections 2024-03-24T11:34:37.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:36.706+0000 7f205f574700 1 -- 172.21.15.114:0/2180337388 wait complete. 2024-03-24T11:34:38.859 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.873+0000 7f88466a5700 1 Processor -- start 2024-03-24T11:34:38.859 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.873+0000 7f88466a5700 1 -- start start 2024-03-24T11:34:38.859 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f88466a5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8840107cc0 con 0x7f8840107100 2024-03-24T11:34:38.859 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883ffff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8840107100 0x7f8840107520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43794/0 (socket says 172.21.15.114:43794) 2024-03-24T11:34:38.859 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883ffff700 1 -- 172.21.15.114:0/4013558824 learned_addr learned my addr 172.21.15.114:0/4013558824 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883effd700 1 -- 172.21.15.114:0/4013558824 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168685380 0 0) 0x7f8840107cc0 con 0x7f8840107100 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883effd700 1 -- 172.21.15.114:0/4013558824 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88280035b0 con 0x7f8840107100 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883effd700 1 -- 172.21.15.114:0/4013558824 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074174665 0 0) 0x7f88280035b0 con 0x7f8840107100 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883effd700 1 -- 172.21.15.114:0/4013558824 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884010b0e0 con 0x7f8840107100 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883effd700 1 -- 172.21.15.114:0/4013558824 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f8834001580 con 0x7f8840107100 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883effd700 1 -- 172.21.15.114:0/4013558824 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f8834001d00 con 0x7f8840107100 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.874+0000 7f883effd700 1 -- 172.21.15.114:0/4013558824 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f8834005960 con 0x7f8840107100 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.875+0000 7f88466a5700 1 -- 172.21.15.114:0/4013558824 >> v1:172.21.15.114:6789/0 conn(0x7f8840107100 legacy=0x7f8840107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.875+0000 7f88466a5700 1 -- 172.21.15.114:0/4013558824 shutdown_connections 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.875+0000 7f88466a5700 1 -- 172.21.15.114:0/4013558824 >> 172.21.15.114:0/4013558824 conn(0x7f88400ff700 msgr2=0x7f8840101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.875+0000 7f88466a5700 1 -- 172.21.15.114:0/4013558824 shutdown_connections 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.875+0000 7f88466a5700 1 -- 172.21.15.114:0/4013558824 wait complete. 2024-03-24T11:34:38.860 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.875+0000 7f88466a5700 1 Processor -- start 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.875+0000 7f88466a5700 1 -- start start 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f88466a5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8840199b50 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883ffff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8840107100 0x7f8840199440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43806/0 (socket says 172.21.15.114:43806) 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883ffff700 1 -- 172.21.15.114:0/3417885161 learned_addr learned my addr 172.21.15.114:0/3417885161 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176901524 0 0) 0x7f8840199b50 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88280054f0 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012577631 0 0) 0x7f88280054f0 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8840199b50 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f8834001080 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 132685073 0 0) 0x7f8840199b50 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.876+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8840199ce0 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.877+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8840199f90 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.877+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f884019dae0 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.877+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f8834004a60 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.877+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f8834006160 con 0x7f8840107100 2024-03-24T11:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.877+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7f8834011590 con 0x7f8840107100 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.878+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f8834047e90 con 0x7f8840107100 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.878+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882c005150 con 0x7f8840107100 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:37.882+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f8834012db0 con 0x7f8840107100 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.079+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 --> v1:172.21.15.114:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f882c005980 con 0x7f8840107100 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.081+0000 7f883d7fa700 1 -- 172.21.15.114:0/3417885161 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (unknown 1496729260 0 0) 0x7f8834014e90 con 0x7f8840107100 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.083+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 >> v1:172.21.15.114:6800/773139778 conn(0x7f8828039f40 legacy=0x7f882803c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.083+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 >> v1:172.21.15.114:6789/0 conn(0x7f8840107100 legacy=0x7f8840199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.083+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 shutdown_connections 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.083+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 >> 172.21.15.114:0/3417885161 conn(0x7f88400ff700 msgr2=0x7f8840101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.083+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 shutdown_connections 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:38.083+0000 7f88466a5700 1 -- 172.21.15.114:0/3417885161 wait complete. 2024-03-24T11:34:38.862 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-03-24T11:34:39.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3417885161' entity='client.admin' 2024-03-24T11:34:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:39 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:40.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3065416357' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:34:40.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:40 smithi114 ceph-mon[30312]: from='mgr.14120 v1:172.21.15.114:0/1427772163' entity='mgr.y' 2024-03-24T11:34:40.435 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:40 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ignoring --setuser ceph since I am not root 2024-03-24T11:34:40.435 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:40 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ignoring --setgroup ceph since I am not root 2024-03-24T11:34:40.709 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:40 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:40.654+0000 7fa90fbd5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:34:40.716 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.305+0000 7f4f5bc78700 1 Processor -- start 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.305+0000 7f4f5bc78700 1 -- start start 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.305+0000 7f4f5bc78700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f541049a0 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.305+0000 7f4f59a14700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4f54103de0 0x7f4f54104200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43812/0 (socket says 172.21.15.114:43812) 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.305+0000 7f4f59a14700 1 -- 172.21.15.114:0/4224603455 learned_addr learned my addr 172.21.15.114:0/4224603455 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.306+0000 7f4f58a12700 1 -- 172.21.15.114:0/4224603455 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214176792 0 0) 0x7f4f541049a0 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.306+0000 7f4f58a12700 1 -- 172.21.15.114:0/4224603455 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f400035b0 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.306+0000 7f4f58a12700 1 -- 172.21.15.114:0/4224603455 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 539618378 0 0) 0x7f4f400035b0 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.306+0000 7f4f58a12700 1 -- 172.21.15.114:0/4224603455 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5410b0a0 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.306+0000 7f4f58a12700 1 -- 172.21.15.114:0/4224603455 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4f50001580 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.306+0000 7f4f58a12700 1 -- 172.21.15.114:0/4224603455 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f4f50001d00 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.306+0000 7f4f58a12700 1 -- 172.21.15.114:0/4224603455 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4f50005960 con 0x7f4f54103de0 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.307+0000 7f4f5bc78700 1 -- 172.21.15.114:0/4224603455 >> v1:172.21.15.114:6789/0 conn(0x7f4f54103de0 legacy=0x7f4f54104200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.307+0000 7f4f5bc78700 1 -- 172.21.15.114:0/4224603455 shutdown_connections 2024-03-24T11:34:40.717 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.307+0000 7f4f5bc78700 1 -- 172.21.15.114:0/4224603455 >> 172.21.15.114:0/4224603455 conn(0x7f4f540ff700 msgr2=0x7f4f54101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.307+0000 7f4f5bc78700 1 -- 172.21.15.114:0/4224603455 shutdown_connections 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.307+0000 7f4f5bc78700 1 -- 172.21.15.114:0/4224603455 wait complete. 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.307+0000 7f4f5bc78700 1 Processor -- start 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.307+0000 7f4f5bc78700 1 -- start start 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f5bc78700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f54199b30 con 0x7f4f54103de0 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f59a14700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4f54103de0 0x7f4f54199420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43820/0 (socket says 172.21.15.114:43820) 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f59a14700 1 -- 172.21.15.114:0/3065416357 learned_addr learned my addr 172.21.15.114:0/3065416357 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924211373 0 0) 0x7f4f54199b30 con 0x7f4f54103de0 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f400054f0 con 0x7f4f54103de0 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1628831150 0 0) 0x7f4f400054f0 con 0x7f4f54103de0 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f54199b30 con 0x7f4f54103de0 2024-03-24T11:34:40.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4f50001080 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3194238589 0 0) 0x7f4f54199b30 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54199cc0 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f54199f70 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5419dac0 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.308+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f4f500049f0 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.309+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f4f50006130 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.309+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 8) v1 ==== 45734+0+0 (unknown 955495283 0 0) 0x7f4f50011620 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.310+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38005150 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.310+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f4f500470b0 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.314+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f4f50016ba0 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:39.571+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4f38005400 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.086+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (unknown 1498667528 0 0) 0x7f4f50014f70 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.087+0000 7f4f4affd700 1 -- 172.21.15.114:0/3065416357 <== mon.0 v1:172.21.15.114:6789/0 11 ==== mgrmap(e 9) v1 ==== 45747+0+0 (unknown 3026381202 0 0) 0x7f4f50045a50 con 0x7f4f54103de0 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.088+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 >> v1:172.21.15.114:6800/773139778 conn(0x7f4f40039ef0 legacy=0x7f4f4003c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:40.719 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.089+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 >> v1:172.21.15.114:6789/0 conn(0x7f4f54103de0 legacy=0x7f4f54199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:40.720 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.089+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 shutdown_connections 2024-03-24T11:34:40.720 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.089+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 >> 172.21.15.114:0/3065416357 conn(0x7f4f540ff700 msgr2=0x7f4f54101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:40.720 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.089+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 shutdown_connections 2024-03-24T11:34:40.720 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:40.089+0000 7f4f5bc78700 1 -- 172.21.15.114:0/3065416357 wait complete. 2024-03-24T11:34:40.966 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:40 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:40.756+0000 7fa90fbd5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:34:41.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3065416357' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:34:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:41 smithi114 ceph-mon[30312]: mgrmap e9: y(active, since 19s) 2024-03-24T11:34:41.686 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:41 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:41.430+0000 7fa90fbd5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:34:41.686 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:41 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:41.543+0000 7fa90fbd5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:34:41.686 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:41 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:41.685+0000 7fa90fbd5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:34:41.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:41 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:41.787+0000 7fa90fbd5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:34:41.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:41 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:41.872+0000 7fa90fbd5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.186+0000 7f45b66ea700 1 Processor -- start 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.186+0000 7f45b66ea700 1 -- start start 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45b66ea700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f45b0079720 con 0x7f45b01065d0 2024-03-24T11:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45affff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f45b01065d0 0x7f45b01089c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54810/0 (socket says 172.21.15.114:54810) 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45affff700 1 -- 172.21.15.114:0/1753140500 learned_addr learned my addr 172.21.15.114:0/1753140500 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45aeffd700 1 -- 172.21.15.114:0/1753140500 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111467698 0 0) 0x7f45b0079720 con 0x7f45b01065d0 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45aeffd700 1 -- 172.21.15.114:0/1753140500 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45980035b0 con 0x7f45b01065d0 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45aeffd700 1 -- 172.21.15.114:0/1753140500 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3752533159 0 0) 0x7f45980035b0 con 0x7f45b01065d0 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45aeffd700 1 -- 172.21.15.114:0/1753140500 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b010b0f0 con 0x7f45b01065d0 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.187+0000 7f45aeffd700 1 -- 172.21.15.114:0/1753140500 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f45a0001580 con 0x7f45b01065d0 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.188+0000 7f45aeffd700 1 -- 172.21.15.114:0/1753140500 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f45a0001d00 con 0x7f45b01065d0 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.188+0000 7f45aeffd700 1 -- 172.21.15.114:0/1753140500 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f45a0005960 con 0x7f45b01065d0 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.188+0000 7f45b66ea700 1 -- 172.21.15.114:0/1753140500 >> v1:172.21.15.114:6789/0 conn(0x7f45b01065d0 legacy=0x7f45b01089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.188+0000 7f45b66ea700 1 -- 172.21.15.114:0/1753140500 shutdown_connections 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.188+0000 7f45b66ea700 1 -- 172.21.15.114:0/1753140500 >> 172.21.15.114:0/1753140500 conn(0x7f45b00fffa0 msgr2=0x7f45b0102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.189+0000 7f45b66ea700 1 -- 172.21.15.114:0/1753140500 shutdown_connections 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.189+0000 7f45b66ea700 1 -- 172.21.15.114:0/1753140500 wait complete. 2024-03-24T11:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.189+0000 7f45b66ea700 1 Processor -- start 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.189+0000 7f45b66ea700 1 -- start start 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45b66ea700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f45b0199b70 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45affff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f45b01065d0 0x7f45b0199460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54820/0 (socket says 172.21.15.114:54820) 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45affff700 1 -- 172.21.15.114:0/1813444731 learned_addr learned my addr 172.21.15.114:0/1813444731 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818642298 0 0) 0x7f45b0199b70 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45980054f0 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388676221 0 0) 0x7f45980054f0 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45b0199b70 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f45a0001080 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4177511090 0 0) 0x7f45b0199b70 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.190+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0199d00 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.191+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f45a00059b0 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.191+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b0199fb0 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.191+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f45a00063b0 con 0x7f45b01065d0 2024-03-24T11:34:42.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.191+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b019db00 con 0x7f45b01065d0 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.192+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b0193170 con 0x7f45b01065d0 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.192+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 9) v1 ==== 45747+0+0 (unknown 3026381202 0 0) 0x7f45a0011950 con 0x7f45b01065d0 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.192+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7f45a0047260 con 0x7f45b01065d0 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.192+0000 7f45af7fe700 1 -- 172.21.15.114:0/1813444731 >> v1:172.21.15.114:6800/773139778 conn(0x7f4598039ef0 legacy=0x7f459803c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/773139778 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.201+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f45a00135d0 con 0x7f45b01065d0 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.393+0000 7f45af7fe700 1 -- 172.21.15.114:0/1813444731 >> v1:172.21.15.114:6800/773139778 conn(0x7f4598039ef0 legacy=0x7f459803c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/773139778 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.463+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f45b002d760 con 0x7f45b01065d0 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.464+0000 7f45ad7fa700 1 -- 172.21.15.114:0/1813444731 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7f45a00150e0 con 0x7f45b01065d0 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.466+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 >> v1:172.21.15.114:6800/773139778 conn(0x7f4598039ef0 legacy=0x7f459803c3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.466+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 >> v1:172.21.15.114:6789/0 conn(0x7f45b01065d0 legacy=0x7f45b0199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.466+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 shutdown_connections 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.466+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 >> 172.21.15.114:0/1813444731 conn(0x7f45b00fffa0 msgr2=0x7f45b01902b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.466+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 shutdown_connections 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:41.466+0000 7f45b66ea700 1 -- 172.21.15.114:0/1813444731 wait complete. 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-03-24T11:34:42.079 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-03-24T11:34:42.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1813444731' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:34:42.324 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:42 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:42.144+0000 7fa90fbd5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:34:42.325 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:42 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:42.254+0000 7fa90fbd5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:42.623 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:42 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:42.347+0000 7fa90fbd5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:34:42.623 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:42 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:42.442+0000 7fa90fbd5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:34:42.623 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:42 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:42.622+0000 7fa90fbd5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:34:43.950 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:43 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:43.598+0000 7fa90fbd5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:34:43.951 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:43 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:43.727+0000 7fa90fbd5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:34:44.238 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:44.063+0000 7fa90fbd5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:34:44.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:44.154+0000 7fa90fbd5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:34:44.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:44.349+0000 7fa90fbd5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:44.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:44.535+0000 7fa90fbd5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:34:45.074 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:44.817+0000 7fa90fbd5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:45.075 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:45 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:45.074+0000 7fa90fbd5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:45.488 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:45 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:45.310+0000 7fa90fbd5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:34:46.489 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:46 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:46.106+0000 7fa90fbd5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:34:46.984 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:46 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:46.670+0000 7fa90fbd5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:34:46.984 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:46 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:46.767+0000 7fa90fbd5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:34:46.984 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:34:46 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:34:46.983+0000 7fa90fbd5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: Active manager daemon y restarted 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: Activating manager daemon y 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: mgrmap e10: y(active, starting, since 0.00321952s) 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: Manager daemon y is now available 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:34:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:34:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:48 smithi114 ceph-mon[30312]: mgrmap e11: y(active, since 1.00651s) 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.542+0000 7fe4f29ab700 1 Processor -- start 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.542+0000 7fe4f29ab700 1 -- start start 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.542+0000 7fe4f29ab700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4ec078e70 con 0x7fe4ec1065d0 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.542+0000 7fe4ebfff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe4ec1065d0 0x7fe4ec1089c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54832/0 (socket says 172.21.15.114:54832) 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.542+0000 7fe4ebfff700 1 -- 172.21.15.114:0/863342559 learned_addr learned my addr 172.21.15.114:0/863342559 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:48.630 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.543+0000 7fe4eaffd700 1 -- 172.21.15.114:0/863342559 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825827323 0 0) 0x7fe4ec078e70 con 0x7fe4ec1065d0 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.543+0000 7fe4eaffd700 1 -- 172.21.15.114:0/863342559 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d80035b0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.543+0000 7fe4eaffd700 1 -- 172.21.15.114:0/863342559 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2606112422 0 0) 0x7fe4d80035b0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.543+0000 7fe4eaffd700 1 -- 172.21.15.114:0/863342559 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec10b0f0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.543+0000 7fe4eaffd700 1 -- 172.21.15.114:0/863342559 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fe4dc001580 con 0x7fe4ec1065d0 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.543+0000 7fe4eaffd700 1 -- 172.21.15.114:0/863342559 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fe4dc001d00 con 0x7fe4ec1065d0 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.543+0000 7fe4eaffd700 1 -- 172.21.15.114:0/863342559 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fe4dc005960 con 0x7fe4ec1065d0 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.544+0000 7fe4f29ab700 1 -- 172.21.15.114:0/863342559 >> v1:172.21.15.114:6789/0 conn(0x7fe4ec1065d0 legacy=0x7fe4ec1089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:48.631 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.544+0000 7fe4f29ab700 1 -- 172.21.15.114:0/863342559 shutdown_connections 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.544+0000 7fe4f29ab700 1 -- 172.21.15.114:0/863342559 >> 172.21.15.114:0/863342559 conn(0x7fe4ec0fff80 msgr2=0x7fe4ec1023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.544+0000 7fe4f29ab700 1 -- 172.21.15.114:0/863342559 shutdown_connections 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.544+0000 7fe4f29ab700 1 -- 172.21.15.114:0/863342559 wait complete. 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4f29ab700 1 Processor -- start 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4f29ab700 1 -- start start 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4f29ab700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4ec199b30 con 0x7fe4ec1065d0 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4ebfff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe4ec1065d0 0x7fe4ec199420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54838/0 (socket says 172.21.15.114:54838) 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4ebfff700 1 -- 172.21.15.114:0/2377853135 learned_addr learned my addr 172.21.15.114:0/2377853135 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395694545 0 0) 0x7fe4ec199b30 con 0x7fe4ec1065d0 2024-03-24T11:34:48.632 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d80054f0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.545+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 403749097 0 0) 0x7fe4d80054f0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4ec199b30 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fe4dc001080 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1138663900 0 0) 0x7fe4ec199b30 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec199cc0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec199f70 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec19dac0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fe4dc0059b0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.546+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fe4dc006260 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.547+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 9) v1 ==== 45747+0+0 (unknown 3026381202 0 0) 0x7fe4dc011740 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.547+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6800/773139778 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4d803dbc0 con 0x7fe4d8039ef0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.547+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 167085054 0 0) 0x7fe4dc048080 con 0x7fe4ec1065d0 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.547+0000 7fe4eb7fe700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6800/773139778 conn(0x7fe4d8039ef0 legacy=0x7fe4d803c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/773139778 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:42.748+0000 7fe4eb7fe700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6800/773139778 conn(0x7fe4d8039ef0 legacy=0x7fe4d803c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/773139778 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:43.149+0000 7fe4eb7fe700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6800/773139778 conn(0x7fe4d8039ef0 legacy=0x7fe4d803c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/773139778 2024-03-24T11:34:48.633 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:43.950+0000 7fe4eb7fe700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6800/773139778 conn(0x7fe4d8039ef0 legacy=0x7fe4d803c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/773139778 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:45.551+0000 7fe4eb7fe700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6800/773139778 conn(0x7fe4d8039ef0 legacy=0x7fe4d803c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/773139778 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:46.999+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mgrmap(e 10) v1 ==== 45549+0+0 (unknown 1441865937 0 0) 0x7fe4dc013580 con 0x7fe4ec1065d0 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:46.999+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6800/773139778 conn(0x7fe4d8039ef0 legacy=0x7fe4d803c3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.001+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mgrmap(e 11) v1 ==== 45641+0+0 (unknown 3465742841 0 0) 0x7fe4dc048cb0 con 0x7fe4ec1065d0 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.001+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6800/3103139641 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4d803dbc0 con 0x7fe4d803ea50 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.002+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (unknown 0 0 4109177037) 0x7fe4d803dbc0 con 0x7fe4d803ea50 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.009+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 --> v1:172.21.15.114:6800/3103139641 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe4cc002630 con 0x7fe4d803ea50 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.009+0000 7fe4e97fa700 1 -- 172.21.15.114:0/2377853135 <== mgr.14152 v1:172.21.15.114:6800/3103139641 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (unknown 0 0 3086460295) 0x7fe4cc002630 con 0x7fe4d803ea50 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.010+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6800/3103139641 conn(0x7fe4d803ea50 legacy=0x7fe4d8040e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.010+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 >> v1:172.21.15.114:6789/0 conn(0x7fe4ec1065d0 legacy=0x7fe4ec199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.010+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 shutdown_connections 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.010+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 >> 172.21.15.114:0/2377853135 conn(0x7fe4ec0fff80 msgr2=0x7fe4ec190270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.010+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 shutdown_connections 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:48.010+0000 7fe4f29ab700 1 -- 172.21.15.114:0/2377853135 wait complete. 2024-03-24T11:34:48.634 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-03-24T11:34:48.635 INFO:teuthology.orchestra.run.smithi114.stdout:Generating a dashboard self-signed certificate... 2024-03-24T11:34:49.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:49 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:48] ENGINE Bus STARTING 2024-03-24T11:34:49.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:49 smithi114 ceph-mon[30312]: from='client.14156 v1:172.21.15.114:0/2377853135' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:34:49.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:49 smithi114 ceph-mon[30312]: from='client.14156 v1:172.21.15.114:0/2377853135' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:34:49.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:49 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:48] ENGINE Serving on https://172.21.15.114:7150 2024-03-24T11:34:49.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:49 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:50.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:50 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:48] ENGINE Serving on http://172.21.15.114:8765 2024-03-24T11:34:50.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:50 smithi114 ceph-mon[30312]: [24/Mar/2024:11:34:48] ENGINE Bus STARTED 2024-03-24T11:34:50.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:50.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:50.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:50 smithi114 ceph-mon[30312]: mgrmap e12: y(active, since 2s) 2024-03-24T11:34:50.412 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T11:34:50.412 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.420+0000 7febc91e5700 1 Processor -- start 2024-03-24T11:34:50.412 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.420+0000 7febc91e5700 1 -- start start 2024-03-24T11:34:50.412 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.420+0000 7febc91e5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febc4079760 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc2d9d700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7febc4106590 0x7febc4108980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54902/0 (socket says 172.21.15.114:54902) 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc2d9d700 1 -- 172.21.15.114:0/2955880493 learned_addr learned my addr 172.21.15.114:0/2955880493 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc1d9b700 1 -- 172.21.15.114:0/2955880493 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103060041 0 0) 0x7febc4079760 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc1d9b700 1 -- 172.21.15.114:0/2955880493 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac0035b0 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc1d9b700 1 -- 172.21.15.114:0/2955880493 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552635588 0 0) 0x7febac0035b0 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc1d9b700 1 -- 172.21.15.114:0/2955880493 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc410b0b0 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc1d9b700 1 -- 172.21.15.114:0/2955880493 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7febb8001580 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc1d9b700 1 -- 172.21.15.114:0/2955880493 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7febb8001c30 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.421+0000 7febc1d9b700 1 -- 172.21.15.114:0/2955880493 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7febb80057e0 con 0x7febc4106590 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.422+0000 7febc91e5700 1 -- 172.21.15.114:0/2955880493 >> v1:172.21.15.114:6789/0 conn(0x7febc4106590 legacy=0x7febc4108980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.422+0000 7febc91e5700 1 -- 172.21.15.114:0/2955880493 shutdown_connections 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.422+0000 7febc91e5700 1 -- 172.21.15.114:0/2955880493 >> 172.21.15.114:0/2955880493 conn(0x7febc40fffa0 msgr2=0x7febc4102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.422+0000 7febc91e5700 1 -- 172.21.15.114:0/2955880493 shutdown_connections 2024-03-24T11:34:50.413 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.422+0000 7febc91e5700 1 -- 172.21.15.114:0/2955880493 wait complete. 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.423+0000 7febc91e5700 1 Processor -- start 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.423+0000 7febc91e5700 1 -- start start 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.423+0000 7febc91e5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febc4199c40 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.423+0000 7febc2d9d700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7febc4106590 0x7febc4199530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54910/0 (socket says 172.21.15.114:54910) 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.423+0000 7febc2d9d700 1 -- 172.21.15.114:0/2659521232 learned_addr learned my addr 172.21.15.114:0/2659521232 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599313397 0 0) 0x7febc4199c40 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac005540 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1483379050 0 0) 0x7febac005540 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc4199c40 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7febb8000ed0 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1800012386 0 0) 0x7febc4199c40 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4199dd0 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc419a080 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febc419dbd0 con 0x7febc4106590 2024-03-24T11:34:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7febb80049d0 con 0x7febc4106590 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.424+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7febb8006160 con 0x7febc4106590 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.425+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 11) v1 ==== 45641+0+0 (unknown 3465742841 0 0) 0x7febb8011540 con 0x7febc4106590 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.425+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7febb8046fb0 con 0x7febc4106590 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.425+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb0005120 con 0x7febc4106590 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.430+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7febb8012e40 con 0x7febc4106590 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.634+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7febb0000b90 con 0x7febac039e70 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.820+0000 7febabfff700 1 -- 172.21.15.114:0/2659521232 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3317752739) 0x7febb0000b90 con 0x7febac039e70 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.822+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 >> v1:172.21.15.114:6800/3103139641 conn(0x7febac039e70 legacy=0x7febac03c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.822+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 >> v1:172.21.15.114:6789/0 conn(0x7febc4106590 legacy=0x7febc4199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.822+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 shutdown_connections 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.822+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 >> 172.21.15.114:0/2659521232 conn(0x7febc40fffa0 msgr2=0x7febc4190290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.822+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 shutdown_connections 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:49.822+0000 7febc91e5700 1 -- 172.21.15.114:0/2659521232 wait complete. 2024-03-24T11:34:50.415 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-03-24T11:34:51.336 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:51 smithi114 ceph-mon[30312]: from='client.14164 v1:172.21.15.114:0/2659521232' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:51.336 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ubIgjrpwQ0XPtjcUoiKk8epTJZnGFBJVESIE4SGPHs/inSCMm1J/e", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711280091, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.888+0000 7f0a1d7d6700 1 Processor -- start 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.888+0000 7f0a1d7d6700 1 -- start start 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.888+0000 7f0a1d7d6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a18108e30 con 0x7f0a18104a00 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.888+0000 7f0a16ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0a18104a00 0x7f0a18104e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58320/0 (socket says 172.21.15.114:58320) 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.888+0000 7f0a16ffd700 1 -- 172.21.15.114:0/1021097955 learned_addr learned my addr 172.21.15.114:0/1021097955 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.888+0000 7f0a15ffb700 1 -- 172.21.15.114:0/1021097955 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206574449 0 0) 0x7f0a18108e30 con 0x7f0a18104a00 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.889+0000 7f0a15ffb700 1 -- 172.21.15.114:0/1021097955 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000035b0 con 0x7f0a18104a00 2024-03-24T11:34:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.889+0000 7f0a15ffb700 1 -- 172.21.15.114:0/1021097955 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2153397156 0 0) 0x7f0a000035b0 con 0x7f0a18104a00 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.889+0000 7f0a15ffb700 1 -- 172.21.15.114:0/1021097955 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18109fd0 con 0x7f0a18104a00 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.889+0000 7f0a15ffb700 1 -- 172.21.15.114:0/1021097955 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a08001580 con 0x7f0a18104a00 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.889+0000 7f0a15ffb700 1 -- 172.21.15.114:0/1021097955 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f0a08001d00 con 0x7f0a18104a00 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.889+0000 7f0a15ffb700 1 -- 172.21.15.114:0/1021097955 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a08005960 con 0x7f0a18104a00 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.890+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/1021097955 >> v1:172.21.15.114:6789/0 conn(0x7f0a18104a00 legacy=0x7f0a18104e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.890+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/1021097955 shutdown_connections 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.890+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/1021097955 >> 172.21.15.114:0/1021097955 conn(0x7f0a180fff80 msgr2=0x7f0a181023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.890+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/1021097955 shutdown_connections 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.890+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/1021097955 wait complete. 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.890+0000 7f0a1d7d6700 1 Processor -- start 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f0a1d7d6700 1 -- start start 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f0a1d7d6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a18078a90 con 0x7f0a18104a00 2024-03-24T11:34:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f0a16ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0a18104a00 0x7f0a1819c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58324/0 (socket says 172.21.15.114:58324) 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f0a16ffd700 1 -- 172.21.15.114:0/3751294460 learned_addr learned my addr 172.21.15.114:0/3751294460 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455123525 0 0) 0x7f0a18078a90 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000054f0 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4199706727 0 0) 0x7f0a000054f0 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a18078a90 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.891+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a08001080 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.892+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 19087563 0 0) 0x7f0a18078a90 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.892+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18078c20 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.892+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a18078ed0 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.892+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1819daa0 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.892+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f0a080049f0 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.892+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f0a08006270 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.893+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 12) v1 ==== 45747+0+0 (unknown 1594766135 0 0) 0x7f0a080116b0 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.893+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7f0a08048000 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.893+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a04005150 con 0x7f0a18104a00 2024-03-24T11:34:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:50.898+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f0a08012e80 con 0x7f0a18104a00 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.114+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0a04002430 con 0x7f0a00039fe0 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.338+0000 7f09fffff700 1 -- 172.21.15.114:0/3751294460 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (unknown 0 0 4195457852) 0x7f0a04002430 con 0x7f0a00039fe0 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.340+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 >> v1:172.21.15.114:6800/3103139641 conn(0x7f0a00039fe0 legacy=0x7f0a0003c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.340+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 >> v1:172.21.15.114:6789/0 conn(0x7f0a18104a00 legacy=0x7f0a1819c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.340+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 shutdown_connections 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.340+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 >> 172.21.15.114:0/3751294460 conn(0x7f0a180fff80 msgr2=0x7f0a181023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.341+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 shutdown_connections 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:51.341+0000 7f0a1d7d6700 1 -- 172.21.15.114:0/3751294460 wait complete. 2024-03-24T11:34:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-03-24T11:34:52.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:52 smithi114 ceph-mon[30312]: from='client.14166 v1:172.21.15.114:0/3751294460' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:52.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:52 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:53.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/844201777' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.889+0000 7fbd03cfb700 1 Processor -- start 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.889+0000 7fbd03cfb700 1 -- start start 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.889+0000 7fbd03cfb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbcfc109f30 con 0x7fbcfc105b00 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd01a97700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbcfc105b00 0x7fbcfc105f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58326/0 (socket says 172.21.15.114:58326) 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd01a97700 1 -- 172.21.15.114:0/3522969424 learned_addr learned my addr 172.21.15.114:0/3522969424 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd00a95700 1 -- 172.21.15.114:0/3522969424 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098345984 0 0) 0x7fbcfc109f30 con 0x7fbcfc105b00 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd00a95700 1 -- 172.21.15.114:0/3522969424 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce80035b0 con 0x7fbcfc105b00 2024-03-24T11:34:53.682 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd00a95700 1 -- 172.21.15.114:0/3522969424 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1681183759 0 0) 0x7fbce80035b0 con 0x7fbcfc105b00 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd00a95700 1 -- 172.21.15.114:0/3522969424 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc10b0d0 con 0x7fbcfc105b00 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd00a95700 1 -- 172.21.15.114:0/3522969424 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fbcf8001580 con 0x7fbcfc105b00 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd00a95700 1 -- 172.21.15.114:0/3522969424 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fbcf8001d00 con 0x7fbcfc105b00 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.890+0000 7fbd00a95700 1 -- 172.21.15.114:0/3522969424 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fbcf8005960 con 0x7fbcfc105b00 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 -- 172.21.15.114:0/3522969424 >> v1:172.21.15.114:6789/0 conn(0x7fbcfc105b00 legacy=0x7fbcfc105f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 -- 172.21.15.114:0/3522969424 shutdown_connections 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 -- 172.21.15.114:0/3522969424 >> 172.21.15.114:0/3522969424 conn(0x7fbcfc1010c0 msgr2=0x7fbcfc1034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 -- 172.21.15.114:0/3522969424 shutdown_connections 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 -- 172.21.15.114:0/3522969424 wait complete. 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 Processor -- start 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 -- start start 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.891+0000 7fbd03cfb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbcfc199b10 con 0x7fbcfc105b00 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbd01a97700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbcfc105b00 0x7fbcfc199400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58328/0 (socket says 172.21.15.114:58328) 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbd01a97700 1 -- 172.21.15.114:0/844201777 learned_addr learned my addr 172.21.15.114:0/844201777 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617557446 0 0) 0x7fbcfc199b10 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce80054f0 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1176574462 0 0) 0x7fbce80054f0 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcfc199b10 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fbcf8001080 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3979305981 0 0) 0x7fbcfc199b10 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc199ca0 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc199f50 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc19daa0 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fbcf8005940 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.892+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fbcf8006230 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.893+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 12) v1 ==== 45747+0+0 (unknown 1594766135 0 0) 0x7fbcf8011730 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.893+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7fbcf8047340 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.893+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce0005150 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:52.898+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fbcf8012ff0 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.095+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fbce0005980 con 0x7fbcfc105b00 2024-03-24T11:34:53.684 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.095+0000 7fbcf2ffd700 1 -- 172.21.15.114:0/844201777 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7fbcf8046c30 con 0x7fbcfc105b00 2024-03-24T11:34:53.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.097+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 >> v1:172.21.15.114:6800/3103139641 conn(0x7fbce8039ef0 legacy=0x7fbce803c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:53.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.097+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 >> v1:172.21.15.114:6789/0 conn(0x7fbcfc105b00 legacy=0x7fbcfc199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:53.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.097+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 shutdown_connections 2024-03-24T11:34:53.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.097+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 >> 172.21.15.114:0/844201777 conn(0x7fbcfc1010c0 msgr2=0x7fbcfc1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:53.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.098+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 shutdown_connections 2024-03-24T11:34:53.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:53.098+0000 7fbd03cfb700 1 -- 172.21.15.114:0/844201777 wait complete. 2024-03-24T11:34:53.688 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:34:53.688 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-03-24T11:34:53.695 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:34:53.695 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-03-24T11:34:53.695 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-03-24T11:34:53.695 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114.front.sepia.ceph.com:8443/ 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: Password: j7dyor0flz 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config directory 2024-03-24T11:34:53.697 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling autotune for osd_memory_target 2024-03-24T11:34:54.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:54 smithi114 ceph-mon[30312]: mgrmap e13: y(active, since 6s) 2024-03-24T11:34:55.021 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.121+0000 7ff0f6663700 1 Processor -- start 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0f6663700 1 -- start start 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0f6663700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0f0079760 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0effff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff0f0106580 0x7ff0f0108970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58330/0 (socket says 172.21.15.114:58330) 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0effff700 1 -- 172.21.15.114:0/2533383448 learned_addr learned my addr 172.21.15.114:0/2533383448 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0eeffd700 1 -- 172.21.15.114:0/2533383448 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375236554 0 0) 0x7ff0f0079760 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0eeffd700 1 -- 172.21.15.114:0/2533383448 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d80035b0 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0eeffd700 1 -- 172.21.15.114:0/2533383448 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1581674623 0 0) 0x7ff0d80035b0 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.122+0000 7ff0eeffd700 1 -- 172.21.15.114:0/2533383448 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f010b0a0 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0eeffd700 1 -- 172.21.15.114:0/2533383448 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff0e0001580 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0eeffd700 1 -- 172.21.15.114:0/2533383448 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ff0e0001d00 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0eeffd700 1 -- 172.21.15.114:0/2533383448 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff0e0005960 con 0x7ff0f0106580 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0f6663700 1 -- 172.21.15.114:0/2533383448 >> v1:172.21.15.114:6789/0 conn(0x7ff0f0106580 legacy=0x7ff0f0108970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0f6663700 1 -- 172.21.15.114:0/2533383448 shutdown_connections 2024-03-24T11:34:55.022 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0f6663700 1 -- 172.21.15.114:0/2533383448 >> 172.21.15.114:0/2533383448 conn(0x7ff0f00fff70 msgr2=0x7ff0f01023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0f6663700 1 -- 172.21.15.114:0/2533383448 shutdown_connections 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.123+0000 7ff0f6663700 1 -- 172.21.15.114:0/2533383448 wait complete. 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.124+0000 7ff0f6663700 1 Processor -- start 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.124+0000 7ff0f6663700 1 -- start start 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.124+0000 7ff0f6663700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0f019f4e0 con 0x7ff0f0106580 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.124+0000 7ff0effff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff0f0106580 0x7ff0f0105e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58336/0 (socket says 172.21.15.114:58336) 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.124+0000 7ff0effff700 1 -- 172.21.15.114:0/2788237250 learned_addr learned my addr 172.21.15.114:0/2788237250 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314129293 0 0) 0x7ff0f019f4e0 con 0x7ff0f0106580 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d80054f0 con 0x7ff0f0106580 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1903544354 0 0) 0x7ff0d80054f0 con 0x7ff0f0106580 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0f019f4e0 con 0x7ff0f0106580 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff0e0001080 con 0x7ff0f0106580 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2661370451 0 0) 0x7ff0f019f4e0 con 0x7ff0f0106580 2024-03-24T11:34:55.023 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f01a0680 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f019f6d0 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f019fbd0 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ff0e00059b0 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.125+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7ff0e0006230 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.126+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7ff0e0011740 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.126+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0dc005120 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.126+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7ff0e00473d0 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.131+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ff0e0016e10 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.325+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 --> v1:172.21.15.114:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7ff0dc005950 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.326+0000 7ff0ed7fa700 1 -- 172.21.15.114:0/2788237250 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (unknown 689008156 0 0) 0x7ff0e0005c30 con 0x7ff0f0106580 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.328+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 >> v1:172.21.15.114:6800/3103139641 conn(0x7ff0d8039f90 legacy=0x7ff0d803c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.328+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 >> v1:172.21.15.114:6789/0 conn(0x7ff0f0106580 legacy=0x7ff0f0105e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.328+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 shutdown_connections 2024-03-24T11:34:55.024 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.328+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 >> 172.21.15.114:0/2788237250 conn(0x7ff0f00fff70 msgr2=0x7ff0f0190280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:55.025 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.328+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 shutdown_connections 2024-03-24T11:34:55.025 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:54.328+0000 7ff0f6663700 1 -- 172.21.15.114:0/2788237250 wait complete. 2024-03-24T11:34:56.236 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.442+0000 7f29349ee700 1 Processor -- start 2024-03-24T11:34:56.236 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.442+0000 7f29349ee700 1 -- start start 2024-03-24T11:34:56.236 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.442+0000 7f29349ee700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2930078ed0 con 0x7f2930106570 2024-03-24T11:34:56.236 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.442+0000 7f292e59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2930106570 0x7f2930108960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58342/0 (socket says 172.21.15.114:58342) 2024-03-24T11:34:56.236 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.442+0000 7f292e59c700 1 -- 172.21.15.114:0/717332530 learned_addr learned my addr 172.21.15.114:0/717332530 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:56.236 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f292dd9b700 1 -- 172.21.15.114:0/717332530 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612980707 0 0) 0x7f2930078ed0 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f292dd9b700 1 -- 172.21.15.114:0/717332530 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291c0035b0 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f292dd9b700 1 -- 172.21.15.114:0/717332530 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2946313771 0 0) 0x7f291c0035b0 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f292dd9b700 1 -- 172.21.15.114:0/717332530 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293010b090 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f292dd9b700 1 -- 172.21.15.114:0/717332530 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f2920001900 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f292dd9b700 1 -- 172.21.15.114:0/717332530 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2920002080 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f292dd9b700 1 -- 172.21.15.114:0/717332530 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f2920005ce0 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.443+0000 7f29349ee700 1 -- 172.21.15.114:0/717332530 >> v1:172.21.15.114:6789/0 conn(0x7f2930106570 legacy=0x7f2930108960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.444+0000 7f29349ee700 1 -- 172.21.15.114:0/717332530 shutdown_connections 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.444+0000 7f29349ee700 1 -- 172.21.15.114:0/717332530 >> 172.21.15.114:0/717332530 conn(0x7f29300fff80 msgr2=0x7f29301023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.444+0000 7f29349ee700 1 -- 172.21.15.114:0/717332530 shutdown_connections 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.444+0000 7f29349ee700 1 -- 172.21.15.114:0/717332530 wait complete. 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.444+0000 7f29349ee700 1 Processor -- start 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f29349ee700 1 -- start start 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f29349ee700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2930199b50 con 0x7f2930106570 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f292e59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2930106570 0x7f2930199440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58356/0 (socket says 172.21.15.114:58356) 2024-03-24T11:34:56.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f292e59c700 1 -- 172.21.15.114:0/4022567220 learned_addr learned my addr 172.21.15.114:0/4022567220 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709847280 0 0) 0x7f2930199b50 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291c0054f0 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154055669 0 0) 0x7f291c0054f0 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2930199b50 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f2920001400 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2891967697 0 0) 0x7f2930199b50 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930199ce0 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2930199f90 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.445+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f293019dae0 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.446+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2920005d30 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.446+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f29200065e0 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.446+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f2920011a30 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.446+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2910005150 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.447+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7f2920047610 con 0x7f2930106570 2024-03-24T11:34:56.238 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.451+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f2920013380 con 0x7f2930106570 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.750+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 --> v1:172.21.15.114:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f2910005980 con 0x7f2930106570 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.753+0000 7f2927fff700 1 -- 172.21.15.114:0/4022567220 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (unknown 1169358022 0 0) 0x7f2920015460 con 0x7f2930106570 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.755+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 >> v1:172.21.15.114:6800/3103139641 conn(0x7f291c039fe0 legacy=0x7f291c03c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.755+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 >> v1:172.21.15.114:6789/0 conn(0x7f2930106570 legacy=0x7f2930199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.755+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 shutdown_connections 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.755+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 >> 172.21.15.114:0/4022567220 conn(0x7f29300fff80 msgr2=0x7f2930190270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.755+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 shutdown_connections 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-03-24T11:34:55.755+0000 7f29349ee700 1 -- 172.21.15.114:0/4022567220 wait complete. 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /sbin/cephadm shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T11:34:56.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /sbin/cephadm shell 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:34:56.240 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-03-24T11:34:56.274 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:34:56.274 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:34:56.274 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:34:56.289 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:34:56.289 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:34:56.289 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:34:56.344 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:34:56.344 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:34:56.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T11:34:56.411 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:34:56.411 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:34:56.411 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:34:56.466 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:34:56.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChYOvg9D3XtfRdVTk6lCNk6Pd8GmaZkIA+S7+s4vTfm1II6DTu5T8/bL9uZuA4p/arWEys7qlCcFK/wNDUTMihuj81pNSW8y3696Uox0IJsfEG17xk9PhLbPdmNLtovNfvCcHobUW4d5ryaPGiBFC0vEKKpTYFCM+7vBoSZxB1OQz2GEG9Wo6EJc09KDx3v2vwH8ExdzVh1sIH97vCKZ78hwe6oi4cEQ9B3n+adhUz97QA/7X+RrxwogZukK4Ub5SB6oxl2bIk/EATMnXEDbqPfwGi8aHldlcpEkFPqUi+TKP0d4GKoYBbjIzOnp4Cq/kH2jnAOx9wWfB0wDCTFKpLeREfUEHYOZuurJztFqu6OhdjUiQJcltw52i6+tOheVULaIhHDBRJ4Rlf/LeHRyDModvI8mjT5xu+lf35Lz4gz+GW5nbjCFX2Fw4k+om1KBcA1I0rkcJFpo3MDg4s0hEvoj9F5p+cL9gEn2tHZOT0AKF3/TuacTchbxEt1s4Rs3M= ceph-412c996a-e9d2-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:34:56.544 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChYOvg9D3XtfRdVTk6lCNk6Pd8GmaZkIA+S7+s4vTfm1II6DTu5T8/bL9uZuA4p/arWEys7qlCcFK/wNDUTMihuj81pNSW8y3696Uox0IJsfEG17xk9PhLbPdmNLtovNfvCcHobUW4d5ryaPGiBFC0vEKKpTYFCM+7vBoSZxB1OQz2GEG9Wo6EJc09KDx3v2vwH8ExdzVh1sIH97vCKZ78hwe6oi4cEQ9B3n+adhUz97QA/7X+RrxwogZukK4Ub5SB6oxl2bIk/EATMnXEDbqPfwGi8aHldlcpEkFPqUi+TKP0d4GKoYBbjIzOnp4Cq/kH2jnAOx9wWfB0wDCTFKpLeREfUEHYOZuurJztFqu6OhdjUiQJcltw52i6+tOheVULaIhHDBRJ4Rlf/LeHRyDModvI8mjT5xu+lf35Lz4gz+GW5nbjCFX2Fw4k+om1KBcA1I0rkcJFpo3MDg4s0hEvoj9F5p+cL9gEn2tHZOT0AKF3/TuacTchbxEt1s4Rs3M= ceph-412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:34:56.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChYOvg9D3XtfRdVTk6lCNk6Pd8GmaZkIA+S7+s4vTfm1II6DTu5T8/bL9uZuA4p/arWEys7qlCcFK/wNDUTMihuj81pNSW8y3696Uox0IJsfEG17xk9PhLbPdmNLtovNfvCcHobUW4d5ryaPGiBFC0vEKKpTYFCM+7vBoSZxB1OQz2GEG9Wo6EJc09KDx3v2vwH8ExdzVh1sIH97vCKZ78hwe6oi4cEQ9B3n+adhUz97QA/7X+RrxwogZukK4Ub5SB6oxl2bIk/EATMnXEDbqPfwGi8aHldlcpEkFPqUi+TKP0d4GKoYBbjIzOnp4Cq/kH2jnAOx9wWfB0wDCTFKpLeREfUEHYOZuurJztFqu6OhdjUiQJcltw52i6+tOheVULaIhHDBRJ4Rlf/LeHRyDModvI8mjT5xu+lf35Lz4gz+GW5nbjCFX2Fw4k+om1KBcA1I0rkcJFpo3MDg4s0hEvoj9F5p+cL9gEn2tHZOT0AKF3/TuacTchbxEt1s4Rs3M= ceph-412c996a-e9d2-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:34:56.611 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChYOvg9D3XtfRdVTk6lCNk6Pd8GmaZkIA+S7+s4vTfm1II6DTu5T8/bL9uZuA4p/arWEys7qlCcFK/wNDUTMihuj81pNSW8y3696Uox0IJsfEG17xk9PhLbPdmNLtovNfvCcHobUW4d5ryaPGiBFC0vEKKpTYFCM+7vBoSZxB1OQz2GEG9Wo6EJc09KDx3v2vwH8ExdzVh1sIH97vCKZ78hwe6oi4cEQ9B3n+adhUz97QA/7X+RrxwogZukK4Ub5SB6oxl2bIk/EATMnXEDbqPfwGi8aHldlcpEkFPqUi+TKP0d4GKoYBbjIzOnp4Cq/kH2jnAOx9wWfB0wDCTFKpLeREfUEHYOZuurJztFqu6OhdjUiQJcltw52i6+tOheVULaIhHDBRJ4Rlf/LeHRyDModvI8mjT5xu+lf35Lz4gz+GW5nbjCFX2Fw4k+om1KBcA1I0rkcJFpo3MDg4s0hEvoj9F5p+cL9gEn2tHZOT0AKF3/TuacTchbxEt1s4Rs3M= ceph-412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:34:56.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:34:56.867 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:56.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4022567220' entity='client.admin' 2024-03-24T11:34:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.156+0000 7f165ab60700 1 -- 172.21.15.114:0/3871702063 >> v1:172.21.15.114:6789/0 conn(0x7f16541015d0 legacy=0x7f16541019f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.156+0000 7f165ab60700 1 -- 172.21.15.114:0/3871702063 shutdown_connections 2024-03-24T11:34:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.156+0000 7f165ab60700 1 -- 172.21.15.114:0/3871702063 >> 172.21.15.114:0/3871702063 conn(0x7f16540fcb70 msgr2=0x7f16540fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.156+0000 7f165ab60700 1 -- 172.21.15.114:0/3871702063 shutdown_connections 2024-03-24T11:34:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.156+0000 7f165ab60700 1 -- 172.21.15.114:0/3871702063 wait complete. 2024-03-24T11:34:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f165ab60700 1 Processor -- start 2024-03-24T11:34:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f165ab60700 1 -- start start 2024-03-24T11:34:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f165ab60700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1654196570 con 0x7f16541015d0 2024-03-24T11:34:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f16588fc700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f16541015d0 0x7f1654195e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58374/0 (socket says 172.21.15.114:58374) 2024-03-24T11:34:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f16588fc700 1 -- 172.21.15.114:0/954838426 learned_addr learned my addr 172.21.15.114:0/954838426 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:34:59.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488734886 0 0) 0x7f1654196570 con 0x7f16541015d0 2024-03-24T11:34:59.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16440054f0 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3178409627 0 0) 0x7f16440054f0 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1654196570 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.157+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f1648004440 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.158+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2536027003 0 0) 0x7f1654196570 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.158+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1654196700 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.158+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16541969b0 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.158+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f1648001620 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.158+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f1648004fd0 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.158+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f165419a4b0 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.158+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f16480052a0 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.159+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7f16480480a0 con 0x7f16541015d0 2024-03-24T11:34:59.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.159+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1654108520 con 0x7f16541015d0 2024-03-24T11:34:59.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.164+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f1648016bd0 con 0x7f16541015d0 2024-03-24T11:34:59.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.366+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 --> v1:172.21.15.114:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f1654107cb0 con 0x7f16541015d0 2024-03-24T11:34:59.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:59.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:59.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:34:59.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:59.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:59.368 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:34:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:34:59.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.369+0000 7f1651ffb700 1 -- 172.21.15.114:0/954838426 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (unknown 3028693289 0 0) 0x7f1648014fa0 con 0x7f16541015d0 2024-03-24T11:34:59.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.372+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 >> v1:172.21.15.114:6800/3103139641 conn(0x7f1644039fc0 legacy=0x7f164403c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:59.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.372+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 >> v1:172.21.15.114:6789/0 conn(0x7f16541015d0 legacy=0x7f1654195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:34:59.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.372+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 shutdown_connections 2024-03-24T11:34:59.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.372+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 >> 172.21.15.114:0/954838426 conn(0x7f16540fcb70 msgr2=0x7f16540fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:34:59.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.372+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 shutdown_connections 2024-03-24T11:34:59.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:34:59.372+0000 7f165ab60700 1 -- 172.21.15.114:0/954838426 wait complete. 2024-03-24T11:34:59.981 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:34:59.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:35:00.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/954838426' entity='client.admin' 2024-03-24T11:35:00.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:00 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:00.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:00 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:00.661 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.141+0000 7f14a0d06700 1 -- 172.21.15.114:0/1328751239 >> v1:172.21.15.114:6789/0 conn(0x7f149c075ca0 legacy=0x7f149c074100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.141+0000 7f14a0d06700 1 -- 172.21.15.114:0/1328751239 shutdown_connections 2024-03-24T11:35:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.141+0000 7f14a0d06700 1 -- 172.21.15.114:0/1328751239 >> 172.21.15.114:0/1328751239 conn(0x7f149c0fcb70 msgr2=0x7f149c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.141+0000 7f14a0d06700 1 -- 172.21.15.114:0/1328751239 shutdown_connections 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.141+0000 7f14a0d06700 1 -- 172.21.15.114:0/1328751239 wait complete. 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f14a0d06700 1 Processor -- start 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f14a0d06700 1 -- start start 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f14a0d06700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f149c198780 con 0x7f149c075ca0 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f149a59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f149c075ca0 0x7f149c198070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52976/0 (socket says 172.21.15.114:52976) 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f149a59c700 1 -- 172.21.15.114:0/2901863915 learned_addr learned my addr 172.21.15.114:0/2901863915 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071880539 0 0) 0x7f149c198780 con 0x7f149c075ca0 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840054f0 con 0x7f149c075ca0 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 94190005 0 0) 0x7f14840054f0 con 0x7f149c075ca0 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f149c198780 con 0x7f149c075ca0 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f148c0044b0 con 0x7f149c075ca0 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2174535968 0 0) 0x7f149c198780 con 0x7f149c075ca0 2024-03-24T11:35:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c198910 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149c198bc0 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.142+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c19c710 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.143+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f148c001620 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.143+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f148c004ee0 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.143+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f148c005100 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.143+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c005150 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.143+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7f148c047350 con 0x7f149c075ca0 2024-03-24T11:35:02.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.148+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f148c012ee0 con 0x7f149c075ca0 2024-03-24T11:35:02.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.344+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f147c000bc0 con 0x7f1484039fa0 2024-03-24T11:35:02.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.348+0000 7f148b7fe700 1 -- 172.21.15.114:0/2901863915 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f147c000bc0 con 0x7f1484039fa0 2024-03-24T11:35:02.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.350+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 >> v1:172.21.15.114:6800/3103139641 conn(0x7f1484039fa0 legacy=0x7f148403c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.350+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 >> v1:172.21.15.114:6789/0 conn(0x7f149c075ca0 legacy=0x7f149c198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.350+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 shutdown_connections 2024-03-24T11:35:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.350+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 >> 172.21.15.114:0/2901863915 conn(0x7f149c0fcb70 msgr2=0x7f149c107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.350+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 shutdown_connections 2024-03-24T11:35:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:02.350+0000 7f14a0d06700 1 -- 172.21.15.114:0/2901863915 wait complete. 2024-03-24T11:35:03.112 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi119 2024-03-24T11:35:03.112 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:35:03.112 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:35:03.132 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:35:03.132 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:35:03.190 INFO:tasks.cephadm:Adding host smithi119 to orchestrator... 2024-03-24T11:35:03.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch host add smithi119 2024-03-24T11:35:03.427 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:03.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='client.14176 v1:172.21.15.114:0/2901863915' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:03.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.883+0000 7f6d15d28700 1 -- 172.21.15.114:0/464145341 >> v1:172.21.15.114:6789/0 conn(0x7f6d101064f0 legacy=0x7f6d101068d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.883+0000 7f6d15d28700 1 -- 172.21.15.114:0/464145341 shutdown_connections 2024-03-24T11:35:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.883+0000 7f6d15d28700 1 -- 172.21.15.114:0/464145341 >> 172.21.15.114:0/464145341 conn(0x7f6d100fbf30 msgr2=0x7f6d100fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.883+0000 7f6d15d28700 1 -- 172.21.15.114:0/464145341 shutdown_connections 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.884+0000 7f6d15d28700 1 -- 172.21.15.114:0/464145341 wait complete. 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.884+0000 7f6d15d28700 1 Processor -- start 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.884+0000 7f6d15d28700 1 -- start start 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.884+0000 7f6d15d28700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d10198a70 con 0x7f6d101064f0 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.884+0000 7f6d0f7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6d101064f0 0x7f6d10198360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52992/0 (socket says 172.21.15.114:52992) 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0f7fe700 1 -- 172.21.15.114:0/2905876219 learned_addr learned my addr 172.21.15.114:0/2905876219 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509196675 0 0) 0x7f6d10198a70 con 0x7f6d101064f0 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf80054f0 con 0x7f6d101064f0 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2941678049 0 0) 0x7f6cf80054f0 con 0x7f6d101064f0 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d10198a70 con 0x7f6d101064f0 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6d00001980 con 0x7f6d101064f0 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1316581289 0 0) 0x7f6d10198a70 con 0x7f6d101064f0 2024-03-24T11:35:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10198c00 con 0x7f6d101064f0 2024-03-24T11:35:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d10198eb0 con 0x7f6d101064f0 2024-03-24T11:35:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d101a1090 con 0x7f6d101064f0 2024-03-24T11:35:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f6d00001200 con 0x7f6d101064f0 2024-03-24T11:35:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.885+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f6d00004e90 con 0x7f6d101064f0 2024-03-24T11:35:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.886+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f6d00005010 con 0x7f6d101064f0 2024-03-24T11:35:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.886+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cfc005120 con 0x7f6d101064f0 2024-03-24T11:35:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.886+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7f6d000471b0 con 0x7f6d101064f0 2024-03-24T11:35:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:04.891+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f6d00005400 con 0x7f6d101064f0 2024-03-24T11:35:05.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:05.090+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cfc000b90 con 0x7f6cf803a010 2024-03-24T11:35:05.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:05 smithi114 ceph-mon[30312]: from='client.14178 v1:172.21.15.114:0/2905876219' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:06 smithi114 ceph-mon[30312]: Deploying cephadm binary to smithi119 2024-03-24T11:35:08.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:08 smithi114 ceph-mon[30312]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:09.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:09.386+0000 7f6d0cff9700 1 -- 172.21.15.114:0/2905876219 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 2149682571) 0x7f6cfc000b90 con 0x7f6cf803a010 2024-03-24T11:35:09.388 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi119' with addr '172.21.15.119' 2024-03-24T11:35:09.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:09.389+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 >> v1:172.21.15.114:6800/3103139641 conn(0x7f6cf803a010 legacy=0x7f6cf803c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:09.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:09.389+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 >> v1:172.21.15.114:6789/0 conn(0x7f6d101064f0 legacy=0x7f6d10198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:09.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:09.389+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 shutdown_connections 2024-03-24T11:35:09.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:09.389+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 >> 172.21.15.114:0/2905876219 conn(0x7f6d100fbf30 msgr2=0x7f6d10069a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:09.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:09.389+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 shutdown_connections 2024-03-24T11:35:09.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:09.389+0000 7f6d15d28700 1 -- 172.21.15.114:0/2905876219 wait complete. 2024-03-24T11:35:09.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:35:10.221 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:10.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:10 smithi114 ceph-mon[30312]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:10.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:10 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:10.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:10 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:10.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:10 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:11.327 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:11 smithi114 ceph-mon[30312]: Added host smithi119 2024-03-24T11:35:11.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:11.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.756+0000 7f4548300700 1 -- 172.21.15.114:0/2179878014 >> v1:172.21.15.114:6789/0 conn(0x7f45401064f0 legacy=0x7f45401068d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.756+0000 7f4548300700 1 -- 172.21.15.114:0/2179878014 shutdown_connections 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.756+0000 7f4548300700 1 -- 172.21.15.114:0/2179878014 >> 172.21.15.114:0/2179878014 conn(0x7f45400fbf50 msgr2=0x7f45400fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.756+0000 7f4548300700 1 -- 172.21.15.114:0/2179878014 shutdown_connections 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.756+0000 7f4548300700 1 -- 172.21.15.114:0/2179878014 wait complete. 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.757+0000 7f4548300700 1 Processor -- start 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.757+0000 7f4548300700 1 -- start start 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.757+0000 7f4548300700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4540196880 con 0x7f45401064f0 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.757+0000 7f454609c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f45401064f0 0x7f4540196170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56688/0 (socket says 172.21.15.114:56688) 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.757+0000 7f454609c700 1 -- 172.21.15.114:0/1198909919 learned_addr learned my addr 172.21.15.114:0/1198909919 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.757+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786585917 0 0) 0x7f4540196880 con 0x7f45401064f0 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.757+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f452c0054f0 con 0x7f45401064f0 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2471729736 0 0) 0x7f452c0054f0 con 0x7f45401064f0 2024-03-24T11:35:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4540196880 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f453c001980 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3484576911 0 0) 0x7f4540196880 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540196a10 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4540196cc0 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f454019eea0 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f453c001200 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.758+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f453c004e90 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.759+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f453c005010 con 0x7f45401064f0 2024-03-24T11:35:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.759+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4524005120 con 0x7f45401064f0 2024-03-24T11:35:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.759+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7f453c0472c0 con 0x7f45401064f0 2024-03-24T11:35:11.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.764+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f453c005400 con 0x7f45401064f0 2024-03-24T11:35:11.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.962+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4524000b90 con 0x7f452c03a010 2024-03-24T11:35:11.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.963+0000 7f45377fe700 1 -- 172.21.15.114:0/1198909919 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (unknown 0 0 3044644231) 0x7f4524000b90 con 0x7f452c03a010 2024-03-24T11:35:11.964 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:35:11.964 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}] 2024-03-24T11:35:11.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.965+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 >> v1:172.21.15.114:6800/3103139641 conn(0x7f452c03a010 legacy=0x7f452c03c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:11.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.965+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 >> v1:172.21.15.114:6789/0 conn(0x7f45401064f0 legacy=0x7f4540196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:11.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.965+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 shutdown_connections 2024-03-24T11:35:11.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.965+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 >> 172.21.15.114:0/1198909919 conn(0x7f45400fbf50 msgr2=0x7f4540069aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:11.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.965+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 shutdown_connections 2024-03-24T11:35:11.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:11.965+0000 7f4548300700 1 -- 172.21.15.114:0/1198909919 wait complete. 2024-03-24T11:35:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:12 smithi114 ceph-mon[30312]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:12 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:12.515 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:35:12.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:35:12.747 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:13.201 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:13 smithi114 ceph-mon[30312]: from='client.14180 v1:172.21.15.114:0/1198909919' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:14 smithi114 ceph-mon[30312]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:14.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.332+0000 7f53c416f700 1 -- 172.21.15.114:0/2728044004 >> v1:172.21.15.114:6789/0 conn(0x7f53bc1027c0 legacy=0x7f53bc102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.332+0000 7f53c416f700 1 -- 172.21.15.114:0/2728044004 shutdown_connections 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.332+0000 7f53c416f700 1 -- 172.21.15.114:0/2728044004 >> 172.21.15.114:0/2728044004 conn(0x7f53bc0fbf30 msgr2=0x7f53bc0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.332+0000 7f53c416f700 1 -- 172.21.15.114:0/2728044004 shutdown_connections 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.332+0000 7f53c416f700 1 -- 172.21.15.114:0/2728044004 wait complete. 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53c416f700 1 Processor -- start 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53c416f700 1 -- start start 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53c416f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53bc196920 con 0x7f53bc1027c0 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53c1f0b700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f53bc1027c0 0x7f53bc196210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56708/0 (socket says 172.21.15.114:56708) 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53c1f0b700 1 -- 172.21.15.114:0/200656303 learned_addr learned my addr 172.21.15.114:0/200656303 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471038807 0 0) 0x7f53bc196920 con 0x7f53bc1027c0 2024-03-24T11:35:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a80054f0 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.333+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403121058 0 0) 0x7f53a80054f0 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53bc196920 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f53b40044b0 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3617099896 0 0) 0x7f53bc196920 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc196ab0 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc196d60 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc19ef40 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f53b4001620 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f53b4004e30 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.334+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f53b4004fb0 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.335+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a0005120 con 0x7f53bc1027c0 2024-03-24T11:35:14.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.335+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 88575528 0 0) 0x7f53b4046ff0 con 0x7f53bc1027c0 2024-03-24T11:35:14.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.340+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f53b4016800 con 0x7f53bc1027c0 2024-03-24T11:35:14.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:14.539+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f53a00053d0 con 0x7f53bc1027c0 2024-03-24T11:35:15.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:15.018+0000 7f53b2ffd700 1 -- 172.21.15.114:0/200656303 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (unknown 3126668360 0 0) 0x7f53b4014e00 con 0x7f53bc1027c0 2024-03-24T11:35:15.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:15.021+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 >> v1:172.21.15.114:6800/3103139641 conn(0x7f53a803a060 legacy=0x7f53a803c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:15.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:15.021+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 >> v1:172.21.15.114:6789/0 conn(0x7f53bc1027c0 legacy=0x7f53bc196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:15.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:15.021+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 shutdown_connections 2024-03-24T11:35:15.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:15.021+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 >> 172.21.15.114:0/200656303 conn(0x7f53bc0fbf30 msgr2=0x7f53bc107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:15.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:15.021+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 shutdown_connections 2024-03-24T11:35:15.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:15.021+0000 7f53c416f700 1 -- 172.21.15.114:0/200656303 wait complete. 2024-03-24T11:35:15.022 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-03-24T11:35:15.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/200656303' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:35:15.548 INFO:tasks.cephadm:Adding mon.a on smithi114 2024-03-24T11:35:15.548 INFO:tasks.cephadm:Adding mon.c on smithi114 2024-03-24T11:35:15.548 INFO:tasks.cephadm:Adding mon.b on smithi119 2024-03-24T11:35:15.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b' 2024-03-24T11:35:15.782 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/200656303' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:17.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.405+0000 7f8041452700 1 -- 172.21.15.119:0/993372571 >> v1:172.21.15.114:6789/0 conn(0x7f803c1064e0 legacy=0x7f803c1068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:17.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.405+0000 7f8041452700 1 -- 172.21.15.119:0/993372571 shutdown_connections 2024-03-24T11:35:17.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.405+0000 7f8041452700 1 -- 172.21.15.119:0/993372571 >> 172.21.15.119:0/993372571 conn(0x7f803c0fbf60 msgr2=0x7f803c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:17.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.405+0000 7f8041452700 1 -- 172.21.15.119:0/993372571 shutdown_connections 2024-03-24T11:35:17.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.405+0000 7f8041452700 1 -- 172.21.15.119:0/993372571 wait complete. 2024-03-24T11:35:17.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.406+0000 7f8041452700 1 Processor -- start 2024-03-24T11:35:17.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.406+0000 7f8041452700 1 -- start start 2024-03-24T11:35:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.406+0000 7f8041452700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f803c06a800 con 0x7f803c1064e0 2024-03-24T11:35:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.406+0000 7f803affd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f803c1064e0 0x7f803c194dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.119:57796/0 (socket says 172.21.15.119:57796) 2024-03-24T11:35:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f803affd700 1 -- 172.21.15.119:0/603284573 learned_addr learned my addr 172.21.15.119:0/603284573 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:35:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183772246 0 0) 0x7f803c06a800 con 0x7f803c1064e0 2024-03-24T11:35:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80240054f0 con 0x7f803c1064e0 2024-03-24T11:35:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518401976 0 0) 0x7f80240054f0 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f803c06a800 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f8030001980 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2320979679 0 0) 0x7f803c06a800 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c1954d0 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c195780 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c195c00 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f8030001200 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.407+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7f8030004e90 con 0x7f803c1064e0 2024-03-24T11:35:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.408+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f8030005010 con 0x7f803c1064e0 2024-03-24T11:35:17.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.408+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8028005120 con 0x7f803c1064e0 2024-03-24T11:35:17.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.408+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 352014572 0 0) 0x7f8030047420 con 0x7f803c1064e0 2024-03-24T11:35:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.413+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f8030005400 con 0x7f803c1064e0 2024-03-24T11:35:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:17 smithi114 ceph-mon[30312]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:35:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:17 smithi114 ceph-mon[30312]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:17 smithi114 ceph-mon[30312]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:35:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:17 smithi114 ceph-mon[30312]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:35:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:17 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:17 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:17 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:17.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.614+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f8028000c70 con 0x7f802403a010 2024-03-24T11:35:17.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.624+0000 7f8023fff700 1 -- 172.21.15.119:0/603284573 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f8028000c70 con 0x7f802403a010 2024-03-24T11:35:17.624 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled mon update... 2024-03-24T11:35:17.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.625+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 >> v1:172.21.15.114:6800/3103139641 conn(0x7f802403a010 legacy=0x7f802403c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:17.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.625+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 >> v1:172.21.15.114:6789/0 conn(0x7f803c1064e0 legacy=0x7f803c194dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:17.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.625+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 shutdown_connections 2024-03-24T11:35:17.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.625+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 >> 172.21.15.119:0/603284573 conn(0x7f803c0fbf60 msgr2=0x7f803c0fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:17.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.625+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 shutdown_connections 2024-03-24T11:35:17.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:17.625+0000 7f8041452700 1 -- 172.21.15.119:0/603284573 wait complete. 2024-03-24T11:35:18.202 DEBUG:teuthology.orchestra.run.smithi114:mon.c> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T11:35:18.204 DEBUG:teuthology.orchestra.run.smithi119:mon.b> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T11:35:18.207 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:35:18.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:35:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:18.931 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:19 smithi114 ceph-mon[30312]: from='client.14184 v1:172.21.15.119:0/603284573' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:19 smithi114 ceph-mon[30312]: Saving service mon spec with placement smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b;count:3 2024-03-24T11:35:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:19 smithi114 ceph-mon[30312]: Deploying daemon mon.b on smithi119 2024-03-24T11:35:20.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:20 smithi114 ceph-mon[30312]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:22.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.001+0000 7fce12fd1700 1 -- 172.21.15.119:0/3858705925 >> v1:172.21.15.114:6789/0 conn(0x7fce0c107640 legacy=0x7fce0c107a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:22.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.001+0000 7fce12fd1700 1 -- 172.21.15.119:0/3858705925 shutdown_connections 2024-03-24T11:35:22.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.001+0000 7fce12fd1700 1 -- 172.21.15.119:0/3858705925 >> 172.21.15.119:0/3858705925 conn(0x7fce0c0fd060 msgr2=0x7fce0c0ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:22.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.001+0000 7fce12fd1700 1 -- 172.21.15.119:0/3858705925 shutdown_connections 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.001+0000 7fce12fd1700 1 -- 172.21.15.119:0/3858705925 wait complete. 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.002+0000 7fce12fd1700 1 Processor -- start 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.002+0000 7fce12fd1700 1 -- start start 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.002+0000 7fce12fd1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fce0c075e60 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.002+0000 7fce10d6d700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fce0c107640 0x7fce0c073f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.119:37898/0 (socket says 172.21.15.119:37898) 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.002+0000 7fce10d6d700 1 -- 172.21.15.119:0/819112301 learned_addr learned my addr 172.21.15.119:0/819112301 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129110470 0 0) 0x7fce0c075e60 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdfc0054f0 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1972972508 0 0) 0x7fcdfc0054f0 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce0c075e60 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fce00001980 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3953931269 0 0) 0x7fce0c075e60 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c074680 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c074930 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fce00001620 con 0x7fce0c107640 2024-03-24T11:35:22.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 590634015 0 0) 0x7fce00004f40 con 0x7fce0c107640 2024-03-24T11:35:22.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.003+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c074db0 con 0x7fce0c107640 2024-03-24T11:35:22.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.004+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7fce00005180 con 0x7fce0c107640 2024-03-24T11:35:22.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.004+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c102cb0 con 0x7fce0c107640 2024-03-24T11:35:22.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.004+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 352014572 0 0) 0x7fce000471e0 con 0x7fce0c107640 2024-03-24T11:35:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.009+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fce00016860 con 0x7fce0c107640 2024-03-24T11:35:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.305+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce0c0632a0 con 0x7fce0c107640 2024-03-24T11:35:22.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.306+0000 7fce09ffb700 1 -- 172.21.15.119:0/819112301 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+697 (unknown 2237029548 0 830111317) 0x7fce00014e60 con 0x7fce0c107640 2024-03-24T11:35:22.307 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:35:22.307 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":1,"fsid":"412c996a-e9d2-11ee-95cd-87774f69a715","modified":"2024-03-24T11:33:49.159143Z","created":"2024-03-24T11:33:49.159143Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:35:22.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.308+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 >> v1:172.21.15.114:6800/3103139641 conn(0x7fcdfc03a010 legacy=0x7fcdfc03c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:22.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.308+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 >> v1:172.21.15.114:6789/0 conn(0x7fce0c107640 legacy=0x7fce0c073f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:22.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.308+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 shutdown_connections 2024-03-24T11:35:22.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.308+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 >> 172.21.15.119:0/819112301 conn(0x7fce0c0fd060 msgr2=0x7fce0c108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:22.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.308+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 shutdown_connections 2024-03-24T11:35:22.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:22.308+0000 7fce12fd1700 1 -- 172.21.15.119:0/819112301 wait complete. 2024-03-24T11:35:22.310 INFO:teuthology.orchestra.run.smithi119.stderr:dumped monmap epoch 1 2024-03-24T11:35:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:22 smithi114 ceph-mon[30312]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:22 smithi119 systemd[1]: Starting Ceph mon.b for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:35:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 podman[33676]: 2024-03-24 11:35:23.251918535 +0000 UTC m=+0.111104486 container create fd6a85ca5007277daf365e6cefd9bf947757612e9d94ffc9628990cf94837518 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-24T11:35:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 podman[33676]: 2024-03-24 11:35:23.159366149 +0000 UTC m=+0.018552110 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:35:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/819112301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:23.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 podman[33676]: 2024-03-24 11:35:23.437709759 +0000 UTC m=+0.296895721 container init fd6a85ca5007277daf365e6cefd9bf947757612e9d94ffc9628990cf94837518 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 podman[33676]: 2024-03-24 11:35:23.443286212 +0000 UTC m=+0.302472161 container start fd6a85ca5007277daf365e6cefd9bf947757612e9d94ffc9628990cf94837518 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: pidfile_write: ignore empty --pid-file 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: load: jerasure load: lrc 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Git sha 0 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: DB SUMMARY 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: DB Session ID: MZ0RY3X7D9A0TKK3U6U6 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:35:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.env: 0x564470cf7ae0 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.info_log: 0x56447217d920 2024-03-24T11:35:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.statistics: (nil) 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.use_fsync: 0 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:35:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.db_log_dir: 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.wal_dir: 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.write_buffer_manager: 0x564472151400 2024-03-24T11:35:23.809 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.unordered_write: 0 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.row_cache: None 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.wal_filter: None 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:35:23.810 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.wal_compression: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:35:23.811 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_open_files: -1 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:35:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Compression algorithms supported: 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kZSTD supported: 0 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:35:23.813 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.merge_operator: 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_filter: None 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56447217da20) 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:35:23.814 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_type: 0 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: data_block_index_type: 0 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_shortening: 1 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: checksum: 4 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: no_block_cache: 0 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache: 0x564472181090 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_options: 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: capacity : 536870912 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: num_shard_bits : 4 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: strict_capacity_limit : 0 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_compressed: (nil) 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: persistent_cache: (nil) 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_size: 4096 2024-03-24T11:35:23.815 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_size_deviation: 10 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_restart_interval: 16 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_block_restart_interval: 1 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: metadata_block_size: 4096 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: partition_filters: 0 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: use_delta_encoding: 1 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: filter_policy: bloomfilter 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: whole_key_filtering: 1 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: verify_compression: 0 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: format_version: 5 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: enable_index_compression: 1 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_align: 0 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: max_auto_readahead_size: 262144 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: prepopulate_block_cache: 0 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:35:23.816 INFO:journalctl@ceph.mon.b.smithi119.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression: NoCompression 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.num_levels: 7 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:35:23.817 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:35:23.818 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:35:23.819 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:35:23.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.ttl: 2592000 2024-03-24T11:35:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:35:23.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b4017e45-bb97-41ac-9529-caedeec25596 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280123472579, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280123473005, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280123, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b4017e45-bb97-41ac-9529-caedeec25596", "db_session_id": "MZ0RY3X7D9A0TKK3U6U6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280123473090, "job": 1, "event": "recovery_finished"} 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564472272000 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: DB pointer 0x56447225c000 2024-03-24T11:35:23.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** DB Stats ** 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** Compaction Stats [default] ** 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:35:23.824 INFO:journalctl@ceph.mon.b.smithi119.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** Compaction Stats [default] ** 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:35:23.825 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: Block cache BinnedLRUCache@0x564472181090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: using public_addrv v1:172.21.15.119:6789/0 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: starting mon.b rank -1 at public addrs v1:172.21.15.119:6789/0 at bind addrs v1:172.21.15.119:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(???) e0 preinit fsid 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: e1 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:35:23.826 INFO:journalctl@ceph.mon.b.smithi119.stdout: legacy client fscid: -1 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout: No filesystems configured 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='client.14180 v1:172.21.15.114:0/1198909919' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:23.827 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/200656303' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/200656303' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:35:23.828 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='client.14184 v1:172.21.15.119:0/603284573' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: Saving service mon spec with placement smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b;count:3 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: Deploying daemon mon.b on smithi119 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:23.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/819112301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:23.830 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 ceph-mon[33714]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T11:35:23.830 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 bash[33676]: fd6a85ca5007277daf365e6cefd9bf947757612e9d94ffc9628990cf94837518 2024-03-24T11:35:23.830 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:23 smithi119 systemd[1]: Started Ceph mon.b for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:35:24.045 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:35:24.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:35:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.542+0000 7ff950f1b700 1 -- 172.21.15.119:0/2734329987 >> v1:172.21.15.114:6789/0 conn(0x7ff94c106680 legacy=0x7ff94c106a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.542+0000 7ff950f1b700 1 -- 172.21.15.119:0/2734329987 shutdown_connections 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.542+0000 7ff950f1b700 1 -- 172.21.15.119:0/2734329987 >> 172.21.15.119:0/2734329987 conn(0x7ff94c0fc0a0 msgr2=0x7ff94c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.542+0000 7ff950f1b700 1 -- 172.21.15.119:0/2734329987 shutdown_connections 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.542+0000 7ff950f1b700 1 -- 172.21.15.119:0/2734329987 wait complete. 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.544+0000 7ff950f1b700 1 Processor -- start 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.544+0000 7ff950f1b700 1 -- start start 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.544+0000 7ff950f1b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff94c0fd0f0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.544+0000 7ff950f1b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff94c0fd280 con 0x7ff94c106680 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.544+0000 7ff94a59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff94c106680 0x7ff94c0fc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:43210/0 (socket says 172.21.15.119:43210) 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.544+0000 7ff94a59c700 1 -- 172.21.15.119:0/603537576 learned_addr learned my addr 172.21.15.119:0/603537576 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.545+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316367297 0 0) 0x7ff94c0fd0f0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.545+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9340035b0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.545+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581906313 0 0) 0x7ff9340035b0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.545+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff94c0fd0f0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.545+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 2171564034 0 0) 0x7ff9400017e0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.546+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3971998042 0 0) 0x7ff94c0fd0f0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.546+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 >> v1:172.21.15.119:6789/0 conn(0x7ff94c106680 legacy=0x7ff94c0fc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.546+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c1a6c50 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.546+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c1a4c30 con 0x7ff94c0fcc60 2024-03-24T11:35:28.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.546+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c1a50f0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.546+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ff940003b10 con 0x7ff94c0fcc60 2024-03-24T11:35:28.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.546+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 2171564034 0 0) 0x7ff940004990 con 0x7ff94c0fcc60 2024-03-24T11:35:28.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.547+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7ff94000fe00 con 0x7ff94c0fcc60 2024-03-24T11:35:28.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.547+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 352014572 0 0) 0x7ff9400469e0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.547+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c005120 con 0x7ff94c0fcc60 2024-03-24T11:35:28.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.553+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ff9400117a0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: mon.a calling monitor election 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: mon.b calling monitor election 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: monmap e2: 2 mons at {a=v1:172.21.15.114:6789/0,b=v1:172.21.15.119:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: fsmap 2024-03-24T11:35:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:28.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: mgrmap e13: y(active, since 41s) 2024-03-24T11:35:28.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: overall HEALTH_OK 2024-03-24T11:35:28.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:28.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:28.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:28.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:28.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:28.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.828+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff92c0053d0 con 0x7ff94c0fcc60 2024-03-24T11:35:28.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.829+0000 7ff93b7fe700 1 -- 172.21.15.119:0/603537576 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+915 (unknown 3557084514 0 3237050434) 0x7ff940013880 con 0x7ff94c0fcc60 2024-03-24T11:35:28.830 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:35:28.830 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":2,"fsid":"412c996a-e9d2-11ee-95cd-87774f69a715","modified":"2024-03-24T11:35:23.493453Z","created":"2024-03-24T11:33:49.159143Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:35:28.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.831+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 >> v1:172.21.15.114:6800/3103139641 conn(0x7ff93403a170 legacy=0x7ff93403c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:28.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.831+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 >> v1:172.21.15.114:6789/0 conn(0x7ff94c0fcc60 legacy=0x7ff94c1a3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:28.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.831+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 shutdown_connections 2024-03-24T11:35:28.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.831+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 >> 172.21.15.119:0/603537576 conn(0x7ff94c0fc0a0 msgr2=0x7ff94c069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:28.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.831+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 shutdown_connections 2024-03-24T11:35:28.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:28.831+0000 7ff950f1b700 1 -- 172.21.15.119:0/603537576 wait complete. 2024-03-24T11:35:28.832 INFO:teuthology.orchestra.run.smithi119.stderr:dumped monmap epoch 2 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: mon.a calling monitor election 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: mon.b calling monitor election 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: monmap e2: 2 mons at {a=v1:172.21.15.114:6789/0,b=v1:172.21.15.119:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: fsmap 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: mgrmap e13: y(active, since 41s) 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: overall HEALTH_OK 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:28.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:29 smithi119 ceph-mon[33714]: Deploying daemon mon.c on smithi114 2024-03-24T11:35:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/603537576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:29 smithi119 ceph-mon[33714]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:29.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:29 smithi114 ceph-mon[30312]: Deploying daemon mon.c on smithi114 2024-03-24T11:35:29.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/603537576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:29.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:29 smithi114 ceph-mon[30312]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:29.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:30.418 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:35:30.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:35:30.660 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:35:32.049 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 podman[42421]: 2024-03-24 11:35:32.024356669 +0000 UTC m=+0.347478947 container start 6270eb72bb93612557d4b3c6cf3eb2e34df746d87d0e0a2625ab1b4fd3ac636d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-c, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:35:32.049 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:35:32.049 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:35:32.049 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pidfile_write: ignore empty --pid-file 2024-03-24T11:35:32.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[30312]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: load: jerasure load: lrc 2024-03-24T11:35:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:35:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Git sha 0 2024-03-24T11:35:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: DB SUMMARY 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: DB Session ID: 2E53RNDL6GCWS9SCEVE0 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:35:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.env: 0x55f154955ae0 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.info_log: 0x55f15683f8e0 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.statistics: (nil) 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.use_fsync: 0 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:35:32.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.db_log_dir: 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.wal_dir: 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:35:32.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.write_buffer_manager: 0x55f156813400 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:35:32.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.unordered_write: 0 2024-03-24T11:35:32.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:35:32.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:35:32.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:35:32.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:35:32.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.row_cache: None 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.wal_filter: None 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.wal_compression: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:35:32.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:35:32.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:35:32.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_open_files: -1 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:35:32.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Compression algorithms supported: 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kZSTD supported: 0 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.merge_operator: 2024-03-24T11:35:32.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_filter: None 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f15683f9e0) 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_type: 0 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_index_type: 0 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_shortening: 1 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:35:32.500 INFO:journalctl@ceph.mon.c.smithi114.stdout: checksum: 4 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: no_block_cache: 0 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache: 0x55f156843090 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_options: 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: capacity : 536870912 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: num_shard_bits : 4 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: strict_capacity_limit : 0 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_compressed: (nil) 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: persistent_cache: (nil) 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size: 4096 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size_deviation: 10 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_restart_interval: 16 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_block_restart_interval: 1 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: metadata_block_size: 4096 2024-03-24T11:35:32.501 INFO:journalctl@ceph.mon.c.smithi114.stdout: partition_filters: 0 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: use_delta_encoding: 1 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: filter_policy: bloomfilter 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: whole_key_filtering: 1 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: verify_compression: 0 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: format_version: 5 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: enable_index_compression: 1 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_align: 0 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: max_auto_readahead_size: 262144 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: prepopulate_block_cache: 0 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:35:32.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression: NoCompression 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.num_levels: 7 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:35:32.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:35:32.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:35:32.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:35:32.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:35:32.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:35:32.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:35:32.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:35:32.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:35:32.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:35:32.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:35:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.ttl: 2592000 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:35:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 861415c0-f479-4214-92b1-4ba438875b47 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280132054667, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280132055064, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280132, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "861415c0-f479-4214-92b1-4ba438875b47", "db_session_id": "2E53RNDL6GCWS9SCEVE0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280132055144, "job": 1, "event": "recovery_finished"} 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f156934000 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: DB pointer 0x55f15691e000 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** DB Stats ** 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:35:32.510 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-03-24T11:35:32.511 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:35:32.512 INFO:journalctl@ceph.mon.c.smithi114.stdout: Block cache BinnedLRUCache@0x55f156843090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: using public_addrv v1:172.21.15.114:6790/0 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: starting mon.c rank -1 at public addrs v1:172.21.15.114:6790/0 at bind addrs v1:172.21.15.114:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(???) e0 preinit fsid 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: e1 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: legacy client fscid: -1 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-03-24T11:35:32.513 INFO:journalctl@ceph.mon.c.smithi114.stdout: No filesystems configured 2024-03-24T11:35:32.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:35:32.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:35:32.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:35:32.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:35:32.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:35:32.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:35:32.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='client.14180 v1:172.21.15.114:0/1198909919' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/200656303' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/200656303' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='client.14184 v1:172.21.15.119:0/603284573' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: Saving service mon spec with placement smithi114:[v1:172.21.15.114:6789]=a;smithi114:[v1:172.21.15.114:6790]=c;smithi119:[v1:172.21.15.119:6789]=b;count:3 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: Deploying daemon mon.b on smithi119 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/819112301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.a calling monitor election 2024-03-24T11:35:32.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.b calling monitor election 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: monmap e2: 2 mons at {a=v1:172.21.15.114:6789/0,b=v1:172.21.15.119:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: fsmap 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mgrmap e13: y(active, since 41s) 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: overall HEALTH_OK 2024-03-24T11:35:32.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: Deploying daemon mon.c on smithi114 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/603537576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:32.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:32.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 ceph-mon[42466]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-24T11:35:32.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 bash[42421]: 6270eb72bb93612557d4b3c6cf3eb2e34df746d87d0e0a2625ab1b4fd3ac636d 2024-03-24T11:35:32.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:32 smithi114 systemd[1]: Started Ceph mon.c for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:35:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:32 smithi119 ceph-mon[33714]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.138+0000 7f33b48e7700 1 -- 172.21.15.119:0/3076960156 >> v1:172.21.15.114:6789/0 conn(0x7f33900035f0 legacy=0x7f3390005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.138+0000 7f33b48e7700 1 -- 172.21.15.119:0/3076960156 shutdown_connections 2024-03-24T11:35:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.138+0000 7f33b48e7700 1 -- 172.21.15.119:0/3076960156 >> 172.21.15.119:0/3076960156 conn(0x7f33b00fd200 msgr2=0x7f33b00ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.138+0000 7f33b48e7700 1 -- 172.21.15.119:0/3076960156 shutdown_connections 2024-03-24T11:35:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.138+0000 7f33b48e7700 1 -- 172.21.15.119:0/3076960156 wait complete. 2024-03-24T11:35:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.139+0000 7f33b48e7700 1 Processor -- start 2024-03-24T11:35:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.139+0000 7f33b48e7700 1 -- start start 2024-03-24T11:35:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.139+0000 7f33b48e7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33b01a6290 con 0x7f33b0196a40 2024-03-24T11:35:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.139+0000 7f33b48e7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33b01a7470 con 0x7f33b01a25c0 2024-03-24T11:35:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.139+0000 7f33b48e7700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33b01a8670 con 0x7f33900035f0 2024-03-24T11:35:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.140+0000 7f33aed9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f33b01a25c0 0x7f33b01a49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33010/0 (socket says 172.21.15.119:33010) 2024-03-24T11:35:37.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.140+0000 7f33aed9d700 1 -- 172.21.15.119:0/23229605 learned_addr learned my addr 172.21.15.119:0/23229605 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:35:37.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.140+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085284386 0 0) 0x7f33b01a6290 con 0x7f33b0196a40 2024-03-24T11:35:37.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.140+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3398003bb0 con 0x7f33b0196a40 2024-03-24T11:35:37.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.141+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1686044143 0 0) 0x7f3398003bb0 con 0x7f33b0196a40 2024-03-24T11:35:37.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.141+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33b01a6290 con 0x7f33b0196a40 2024-03-24T11:35:37.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.141+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f33a0001830 con 0x7f33b0196a40 2024-03-24T11:35:37.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.141+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1444693974 0 0) 0x7f33b01a6290 con 0x7f33b0196a40 2024-03-24T11:35:37.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.142+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 >> v1:172.21.15.114:6790/0 conn(0x7f33900035f0 legacy=0x7f33b0196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:37.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.142+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 >> v1:172.21.15.119:6789/0 conn(0x7f33b01a25c0 legacy=0x7f33b01a49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:37.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.142+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b01a9870 con 0x7f33b0196a40 2024-03-24T11:35:37.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.142+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b01a8860 con 0x7f33b0196a40 2024-03-24T11:35:37.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.142+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b01a8d20 con 0x7f33b0196a40 2024-03-24T11:35:37.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.142+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f33a0003b10 con 0x7f33b0196a40 2024-03-24T11:35:37.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.142+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f33a00049d0 con 0x7f33b0196a40 2024-03-24T11:35:37.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.143+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f33a000ff00 con 0x7f33b0196a40 2024-03-24T11:35:37.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.143+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3390006b70 con 0x7f33b0196a40 2024-03-24T11:35:37.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.143+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 352014572 0 0) 0x7f33a0045be0 con 0x7f33b0196a40 2024-03-24T11:35:37.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.148+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f33a00118c0 con 0x7f33b0196a40 2024-03-24T11:35:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.438+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3390006e20 con 0x7f33b0196a40 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: mon.a calling monitor election 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: mon.b calling monitor election 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: mon.c calling monitor election 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: monmap e3: 3 mons at {a=v1:172.21.15.114:6789/0,b=v1:172.21.15.119:6789/0,c=v1:172.21.15.114:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: fsmap 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: mgrmap e13: y(active, since 50s) 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: overall HEALTH_OK 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.439+0000 7f339f7fe700 1 -- 172.21.15.119:0/23229605 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1133 (unknown 409899127 0 3996206935) 0x7f33a00139a0 con 0x7f33b0196a40 2024-03-24T11:35:37.441 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-24T11:35:37.442 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":3,"fsid":"412c996a-e9d2-11ee-95cd-87774f69a715","modified":"2024-03-24T11:35:32.074266Z","created":"2024-03-24T11:33:49.159143Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-24T11:35:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.442+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 >> v1:172.21.15.114:6800/3103139641 conn(0x7f3398039f10 legacy=0x7f339803c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.442+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 >> v1:172.21.15.114:6789/0 conn(0x7f33b0196a40 legacy=0x7f33b01a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.442+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 shutdown_connections 2024-03-24T11:35:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.442+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 >> 172.21.15.119:0/23229605 conn(0x7f33b00fd200 msgr2=0x7f33b00fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.442+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 shutdown_connections 2024-03-24T11:35:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:37.442+0000 7f33b48e7700 1 -- 172.21.15.119:0/23229605 wait complete. 2024-03-24T11:35:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:dumped monmap epoch 3 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: mon.a calling monitor election 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: mon.b calling monitor election 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: mon.c calling monitor election 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: monmap e3: 3 mons at {a=v1:172.21.15.114:6789/0,b=v1:172.21.15.119:6789/0,c=v1:172.21.15.114:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: fsmap 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: mgrmap e13: y(active, since 50s) 2024-03-24T11:35:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: overall HEALTH_OK 2024-03-24T11:35:37.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: mon.a calling monitor election 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: mon.b calling monitor election 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: mon.c calling monitor election 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: monmap e3: 3 mons at {a=v1:172.21.15.114:6789/0,b=v1:172.21.15.119:6789/0,c=v1:172.21.15.114:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: fsmap 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: mgrmap e13: y(active, since 50s) 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: overall HEALTH_OK 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:37.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:38.064 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:35:38.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:35:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:38 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/23229605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:38 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:38 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/23229605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:38 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:38 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/23229605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:35:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:38 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:35:38.524 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:39.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:39.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:39.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:39.648 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:39.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:39.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:39.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:40.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:40.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:40.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:40.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.101+0000 7f0ed55ca700 1 -- 172.21.15.114:0/3294733707 >> v1:172.21.15.114:6789/0 conn(0x7f0ed01064f0 legacy=0x7f0ed01068d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:40.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.101+0000 7f0ed55ca700 1 -- 172.21.15.114:0/3294733707 shutdown_connections 2024-03-24T11:35:40.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.101+0000 7f0ed55ca700 1 -- 172.21.15.114:0/3294733707 >> 172.21.15.114:0/3294733707 conn(0x7f0ed00fbf50 msgr2=0x7f0ed00fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:40.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.101+0000 7f0ed55ca700 1 -- 172.21.15.114:0/3294733707 shutdown_connections 2024-03-24T11:35:40.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.101+0000 7f0ed55ca700 1 -- 172.21.15.114:0/3294733707 wait complete. 2024-03-24T11:35:40.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0ed55ca700 1 Processor -- start 2024-03-24T11:35:40.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0ed55ca700 1 -- start start 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0ed55ca700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ed01a62b0 con 0x7f0ed01064f0 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0ed55ca700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ed01a7490 con 0x7f0ed0197d40 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0ed55ca700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ed01a8690 con 0x7f0ed0197850 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0eceffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0ed01064f0 0x7f0ed0197140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55930/0 (socket says 172.21.15.114:55930) 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0eceffd700 1 -- 172.21.15.114:0/4204247360 learned_addr learned my addr 172.21.15.114:0/4204247360 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0ece7fc700 1 --1- 172.21.15.114:0/4204247360 >> v1:172.21.15.114:6790/0 conn(0x7f0ed0197850 0x7f0ed01a11c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39966/0 (socket says 172.21.15.114:39966) 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137138371 0 0) 0x7f0ed01a8690 con 0x7f0ed0197850 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb8003bb0 con 0x7f0ed0197850 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171322152 0 0) 0x7f0ed01a62b0 con 0x7f0ed01064f0 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.102+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed01a8690 con 0x7f0ed01064f0 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479361987 0 0) 0x7f0ed01a7490 con 0x7f0ed0197d40 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed01a62b0 con 0x7f0ed0197d40 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2363662422 0 0) 0x7f0eb8003bb0 con 0x7f0ed0197850 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ed01a7490 con 0x7f0ed0197850 2024-03-24T11:35:40.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0ec4001830 con 0x7f0ed0197850 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2810629875 0 0) 0x7f0ed01a8690 con 0x7f0ed01064f0 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0eb8003bb0 con 0x7f0ed01064f0 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0ec8001c80 con 0x7f0ed01064f0 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2911847899 0 0) 0x7f0ed01a62b0 con 0x7f0ed0197d40 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ed01a8690 con 0x7f0ed0197d40 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0ec0001790 con 0x7f0ed0197d40 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3325257745 0 0) 0x7f0ed01a7490 con 0x7f0ed0197850 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 >> v1:172.21.15.119:6789/0 conn(0x7f0ed0197d40 legacy=0x7f0ed01a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 >> v1:172.21.15.114:6789/0 conn(0x7f0ed01064f0 legacy=0x7f0ed0197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed01a9890 con 0x7f0ed0197850 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed01a64a0 con 0x7f0ed0197850 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed01a6920 con 0x7f0ed0197850 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f0ec4004d00 con 0x7f0ed0197850 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.103+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0ec4005b10 con 0x7f0ed0197850 2024-03-24T11:35:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.104+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f0ec4010f60 con 0x7f0ed0197850 2024-03-24T11:35:40.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.104+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ebc005120 con 0x7f0ed0197850 2024-03-24T11:35:40.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.104+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 352014572 0 0) 0x7f0ec4047ad0 con 0x7f0ed0197850 2024-03-24T11:35:40.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.109+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f0ec4012aa0 con 0x7f0ed0197850 2024-03-24T11:35:40.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.316+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0ebc0053d0 con 0x7f0ed0197850 2024-03-24T11:35:40.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.316+0000 7f0eb7fff700 1 -- 172.21.15.114:0/4204247360 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+193 (unknown 2189217731 0 2184069669) 0x7f0ec40149d0 con 0x7f0ed0197850 2024-03-24T11:35:40.317 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:35:40.317 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-24T11:35:40.317 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:35:40.317 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = 172.21.15.114:6789/0 172.21.15.119:6789/0 v1:172.21.15.114:6790/0 2024-03-24T11:35:40.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.318+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 >> v1:172.21.15.114:6800/3103139641 conn(0x7f0eb803a250 legacy=0x7f0eb803c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:40.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.318+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 >> v1:172.21.15.114:6790/0 conn(0x7f0ed0197850 legacy=0x7f0ed01a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:40.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.318+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 shutdown_connections 2024-03-24T11:35:40.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.318+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 >> 172.21.15.114:0/4204247360 conn(0x7f0ed00fbf50 msgr2=0x7f0ed00fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:40.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.318+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 shutdown_connections 2024-03-24T11:35:40.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:40.318+0000 7f0ed55ca700 1 -- 172.21.15.114:0/4204247360 wait complete. 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[42466]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[42466]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[42466]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[42466]: Reconfiguring daemon mon.a on smithi114 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4204247360' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[30312]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[30312]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[30312]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:35:41.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[30312]: Reconfiguring daemon mon.a on smithi114 2024-03-24T11:35:41.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4204247360' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:40 smithi119 ceph-mon[33714]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:40 smithi119 ceph-mon[33714]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:35:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:40 smithi119 ceph-mon[33714]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:35:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:40 smithi119 ceph-mon[33714]: Reconfiguring daemon mon.a on smithi114 2024-03-24T11:35:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4204247360' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:41.344 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:35:41.344 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:35:41.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:35:41.373 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:35:41.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:35:41.440 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:35:41.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:35:41.475 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:35:41.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:35:41.546 INFO:tasks.cephadm:Adding mgr.y on smithi114 2024-03-24T11:35:41.546 INFO:tasks.cephadm:Adding mgr.x on smithi119 2024-03-24T11:35:41.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi114=y;smithi119=x' 2024-03-24T11:35:41.820 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[30312]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[42466]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:41.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:41.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:42.201 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:41 smithi119 ceph-mon[33714]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:42.201 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:42.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:42.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:42.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:42.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:43.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:43 smithi119 ceph-mon[33714]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:35:43.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:43 smithi119 ceph-mon[33714]: Reconfiguring daemon mon.c on smithi114 2024-03-24T11:35:43.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:43 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:43.333 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:43 smithi114 ceph-mon[42466]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:35:43.333 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:43 smithi114 ceph-mon[42466]: Reconfiguring daemon mon.c on smithi114 2024-03-24T11:35:43.333 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:43 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:43.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:43 smithi114 ceph-mon[30312]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:35:43.334 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:43 smithi114 ceph-mon[30312]: Reconfiguring daemon mon.c on smithi114 2024-03-24T11:35:43.334 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:43 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:43.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- 172.21.15.119:0/289736302 >> v1:172.21.15.114:6789/0 conn(0x7faaf0101710 legacy=0x7faaf0101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- 172.21.15.119:0/289736302 shutdown_connections 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- 172.21.15.119:0/289736302 >> 172.21.15.119:0/289736302 conn(0x7faaf00fccd0 msgr2=0x7faaf00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- 172.21.15.119:0/289736302 shutdown_connections 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- 172.21.15.119:0/289736302 wait complete. 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 Processor -- start 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- start start 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faaf01a1880 con 0x7faaf0196720 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faaf01a2a60 con 0x7faaf019dc10 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.369+0000 7faaf6f7f700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faaf01a3c60 con 0x7faaf0101710 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaf551c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7faaf019dc10 0x7faaf01a0000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51554/0 (socket says 172.21.15.119:51554) 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaf551c700 1 -- 172.21.15.119:0/576779477 learned_addr learned my addr 172.21.15.119:0/576779477 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982707863 0 0) 0x7faaf01a2a60 con 0x7faaf019dc10 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae0003bb0 con 0x7faaf019dc10 2024-03-24T11:35:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730936954 0 0) 0x7faaf01a1880 con 0x7faaf0196720 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf01a2a60 con 0x7faaf0196720 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200736655 0 0) 0x7faaf01a3c60 con 0x7faaf0101710 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf01a1880 con 0x7faaf0101710 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 283921098 0 0) 0x7faae0003bb0 con 0x7faaf019dc10 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faaf01a3c60 con 0x7faaf019dc10 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7faae8001b70 con 0x7faaf019dc10 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1767871739 0 0) 0x7faaf01a2a60 con 0x7faaf0196720 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faae0003bb0 con 0x7faaf0196720 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 520397367 0 0) 0x7faaf01a1880 con 0x7faaf0101710 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faaf01a2a60 con 0x7faaf0101710 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7faadc001810 con 0x7faaf0196720 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7faae4000fe0 con 0x7faaf0101710 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2808464677 0 0) 0x7faaf01a3c60 con 0x7faaf019dc10 2024-03-24T11:35:43.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 >> v1:172.21.15.114:6790/0 conn(0x7faaf0101710 legacy=0x7faaf0196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 >> v1:172.21.15.114:6789/0 conn(0x7faaf0196720 legacy=0x7faaf019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf01a4e60 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf01a1a70 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7faae80051c0 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.370+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7faae8005fe0 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.371+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf01a1f70 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.371+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7faae80115a0 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.371+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 352014572 0 0) 0x7faae8047230 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.371+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf00632a0 con 0x7faaf019dc10 2024-03-24T11:35:43.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.376+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7faae8012eb0 con 0x7faaf019dc10 2024-03-24T11:35:43.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.593+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi119=x", "target": ["mon-mgr", ""]}) v1 -- 0x7faaf01a22c0 con 0x7faae003a370 2024-03-24T11:35:43.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.599+0000 7faaedffb700 1 -- 172.21.15.119:0/576779477 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7faaf01a22c0 con 0x7faae003a370 2024-03-24T11:35:43.600 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled mgr update... 2024-03-24T11:35:43.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.601+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 >> v1:172.21.15.114:6800/3103139641 conn(0x7faae003a370 legacy=0x7faae003c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:43.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.601+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 >> v1:172.21.15.119:6789/0 conn(0x7faaf019dc10 legacy=0x7faaf01a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:43.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.601+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 shutdown_connections 2024-03-24T11:35:43.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.601+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 >> 172.21.15.119:0/576779477 conn(0x7faaf00fccd0 msgr2=0x7faaf00ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:43.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.601+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 shutdown_connections 2024-03-24T11:35:43.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:35:43.601+0000 7faaf6f7f700 1 -- 172.21.15.119:0/576779477 wait complete. 2024-03-24T11:35:44.170 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:44 smithi119 ceph-mon[33714]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:44.170 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:44 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.170 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:44 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.171 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:44 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:44.171 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:44 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:44.171 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:44 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:44.171 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:44 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.196 DEBUG:teuthology.orchestra.run.smithi119:mgr.x> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T11:35:44.199 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:35:44.199 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:35:44.199 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:35:44.219 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:35:44.219 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-03-24T11:35:44.277 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:35:44.277 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:44.277 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T11:35:44.277 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:44.277 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:44.278 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-24 11:35:31.511316805 +0000 2024-03-24T11:35:44.278 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-24 11:30:03.488849986 +0000 2024-03-24T11:35:44.278 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-24 11:30:03.488849986 +0000 2024-03-24T11:35:44.278 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-24 11:30:03.488849986 +0000 2024-03-24T11:35:44.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:35:44.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[30312]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:44.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:44.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:44.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:44.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[42466]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:44.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:35:44.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:35:44.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:44.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:44 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-24T11:35:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-24T11:35:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000202196 s, 2.5 MB/s 2024-03-24T11:35:44.350 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:35:44.409 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-24 11:35:31.511316805 +0000 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-24 11:30:03.827842297 +0000 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-24 11:30:03.827842297 +0000 2024-03-24T11:35:44.467 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-24 11:30:03.827842297 +0000 2024-03-24T11:35:44.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:35:44.536 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-24T11:35:44.536 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-24T11:35:44.536 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000208923 s, 2.5 MB/s 2024-03-24T11:35:44.537 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:35:44.595 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-03-24T11:35:44.652 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-24 11:35:31.511316805 +0000 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-24 11:30:04.158834789 +0000 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-24 11:30:04.158834789 +0000 2024-03-24T11:35:44.653 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-24 11:30:04.158834789 +0000 2024-03-24T11:35:44.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:35:44.720 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-24T11:35:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-24T11:35:44.721 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000201858 s, 2.5 MB/s 2024-03-24T11:35:44.722 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:35:44.779 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-03-24T11:35:44.834 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-24 11:35:31.511316805 +0000 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-24 11:30:04.501827008 +0000 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-24 11:30:04.501827008 +0000 2024-03-24T11:35:44.835 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-24 11:30:04.501827008 +0000 2024-03-24T11:35:44.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:35:44.901 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-24T11:35:44.901 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-24T11:35:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000208046 s, 2.5 MB/s 2024-03-24T11:35:44.903 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:35:44.959 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:35:44.959 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:35:44.976 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:35:44.976 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-03-24T11:35:45.031 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-24 11:35:22.347340377 +0000 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-24 11:30:03.500297654 +0000 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-24 11:30:03.500297654 +0000 2024-03-24T11:35:45.032 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-24 11:30:03.500297654 +0000 2024-03-24T11:35:45.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:35:45.098 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-24T11:35:45.098 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-24T11:35:45.098 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000186348 s, 2.7 MB/s 2024-03-24T11:35:45.100 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:35:45.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:45 smithi119 ceph-mon[33714]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:45 smithi119 ceph-mon[33714]: Reconfiguring daemon mon.b on smithi119 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:45 smithi119 ceph-mon[33714]: from='client.24103 v1:172.21.15.119:0/576779477' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi119=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:45 smithi119 ceph-mon[33714]: Saving service mgr spec with placement smithi114=y;smithi119=x;count:2 2024-03-24T11:35:45.155 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-03-24T11:35:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:35:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:45.212 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T11:35:45.212 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:45.212 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:45.212 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-24 11:35:22.347340377 +0000 2024-03-24T11:35:45.212 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-24 11:30:03.846289272 +0000 2024-03-24T11:35:45.212 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-24 11:30:03.846289272 +0000 2024-03-24T11:35:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-24 11:30:03.846289272 +0000 2024-03-24T11:35:45.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:35:45.277 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-24T11:35:45.278 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-24T11:35:45.278 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000188818 s, 2.7 MB/s 2024-03-24T11:35:45.279 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:35:45.335 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-03-24T11:35:45.390 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:35:45.390 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:45.390 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T11:35:45.390 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:45.390 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:45.390 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-24 11:35:22.348340354 +0000 2024-03-24T11:35:45.391 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-24 11:30:04.211280430 +0000 2024-03-24T11:35:45.391 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-24 11:30:04.211280430 +0000 2024-03-24T11:35:45.391 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-24 11:30:04.211280430 +0000 2024-03-24T11:35:45.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:35:45.456 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-24T11:35:45.456 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-24T11:35:45.456 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000207746 s, 2.5 MB/s 2024-03-24T11:35:45.457 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[30312]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[30312]: Reconfiguring daemon mon.b on smithi119 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[30312]: from='client.24103 v1:172.21.15.119:0/576779477' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi119=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[30312]: Saving service mgr spec with placement smithi114=y;smithi119=x;count:2 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[42466]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[42466]: Reconfiguring daemon mon.b on smithi119 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[42466]: from='client.24103 v1:172.21.15.119:0/576779477' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi119=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:45 smithi114 ceph-mon[42466]: Saving service mgr spec with placement smithi114=y;smithi119=x;count:2 2024-03-24T11:35:45.512 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-24 11:35:22.348340354 +0000 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-24 11:30:04.538272508 +0000 2024-03-24T11:35:45.567 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-24 11:30:04.538272508 +0000 2024-03-24T11:35:45.568 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-24 11:30:04.538272508 +0000 2024-03-24T11:35:45.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:35:45.633 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-24T11:35:45.633 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-24T11:35:45.633 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000198186 s, 2.6 MB/s 2024-03-24T11:35:45.634 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:35:45.690 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-03-24T11:35:45.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:35:45.929 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:35:46.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:35:46.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:35:46.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:47 smithi114 ceph-mon[30312]: Deploying daemon mgr.x on smithi119 2024-03-24T11:35:47.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:47 smithi114 ceph-mon[42466]: Deploying daemon mgr.x on smithi119 2024-03-24T11:35:47.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:47 smithi119 ceph-mon[33714]: Deploying daemon mgr.x on smithi119 2024-03-24T11:35:47.470 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:47 smithi119 systemd[1]: Starting Ceph mgr.x for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:35:47.804 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:47 smithi119 podman[35772]: 2024-03-24 11:35:47.469688943 +0000 UTC m=+0.103706008 container create f224daaff314a729dd72599eb51c79e5d7cc2962cb63f5c85a8a83d0696c0fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T11:35:47.804 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:47 smithi119 podman[35772]: 2024-03-24 11:35:47.383996981 +0000 UTC m=+0.018014044 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:35:47.805 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:47 smithi119 podman[35772]: 2024-03-24 11:35:47.697151997 +0000 UTC m=+0.331169060 container init f224daaff314a729dd72599eb51c79e5d7cc2962cb63f5c85a8a83d0696c0fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T11:35:47.805 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:47 smithi119 podman[35772]: 2024-03-24 11:35:47.702227499 +0000 UTC m=+0.336244561 container start f224daaff314a729dd72599eb51c79e5d7cc2962cb63f5c85a8a83d0696c0fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T11:35:47.815 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:35:47.815 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:35:47.815 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-24T11:35:47.815 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-24T11:35:47.815 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113389 s, 925 MB/s 2024-03-24T11:35:47.815 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-24T11:35:48.075 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:47 smithi119 bash[35772]: f224daaff314a729dd72599eb51c79e5d7cc2962cb63f5c85a8a83d0696c0fbb 2024-03-24T11:35:48.075 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:47 smithi119 systemd[1]: Started Ceph mgr.x for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:35:48.355 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:48.258+0000 7f5e0ef09200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:35:48.356 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:48.354+0000 7f5e0ef09200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:35:48.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-mon[33714]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:48.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:48.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[42466]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:48.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[30312]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:48.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:48.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-03-24T11:35:48.703 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:49.133 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.038+0000 7f5e0ef09200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:35:49.133 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.132+0000 7f5e0ef09200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:35:49.461 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.274+0000 7f5e0ef09200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:35:49.461 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.375+0000 7f5e0ef09200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:35:49.461 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.460+0000 7f5e0ef09200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:35:50.006 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.718+0000 7f5e0ef09200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:35:50.006 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.826+0000 7f5e0ef09200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:35:50.006 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:49 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:49.911+0000 7f5e0ef09200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:35:50.006 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:50.005+0000 7f5e0ef09200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:35:50.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:50.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:50.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:50.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:35:50.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.284+0000 7f00912d2700 1 -- 172.21.15.114:0/3332039481 >> v1:172.21.15.114:6790/0 conn(0x7f008c105eb0 legacy=0x7f008c108300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.284+0000 7f00912d2700 1 -- 172.21.15.114:0/3332039481 shutdown_connections 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.284+0000 7f00912d2700 1 -- 172.21.15.114:0/3332039481 >> 172.21.15.114:0/3332039481 conn(0x7f008c0fd020 msgr2=0x7f008c0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.284+0000 7f00912d2700 1 -- 172.21.15.114:0/3332039481 shutdown_connections 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.284+0000 7f00912d2700 1 -- 172.21.15.114:0/3332039481 wait complete. 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.284+0000 7f00912d2700 1 Processor -- start 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f00912d2700 1 -- start start 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f00912d2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f008c1a3d40 con 0x7f008c109bf0 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f00912d2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f008c1a4f40 con 0x7f008c105eb0 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f00912d2700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f008c1a6140 con 0x7f008c101a80 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f008affd700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f008c101a80 0x7f008c075930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52470/0 (socket says 172.21.15.114:52470) 2024-03-24T11:35:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f008affd700 1 -- 172.21.15.114:0/1097959967 learned_addr learned my addr 172.21.15.114:0/1097959967 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f008b7fe700 1 --1- 172.21.15.114:0/1097959967 >> v1:172.21.15.114:6789/0 conn(0x7f008c109bf0 0x7f008c1a2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48374/0 (socket says 172.21.15.114:48374) 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012056724 0 0) 0x7f008c1a3d40 con 0x7f008c109bf0 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078005070 con 0x7f008c109bf0 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225656403 0 0) 0x7f008c1a6140 con 0x7f008c101a80 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f008c1a3d40 con 0x7f008c101a80 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652955354 0 0) 0x7f008c1a4f40 con 0x7f008c105eb0 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f008c1a6140 con 0x7f008c105eb0 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 696962904 0 0) 0x7f008c1a3d40 con 0x7f008c101a80 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f008c1a4f40 con 0x7f008c101a80 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915479678 0 0) 0x7f0078005070 con 0x7f008c109bf0 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f008c1a3d40 con 0x7f008c109bf0 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f007c001980 con 0x7f008c101a80 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2697880884 0 0) 0x7f008c1a6140 con 0x7f008c105eb0 2024-03-24T11:35:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0078005070 con 0x7f008c105eb0 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0084001c50 con 0x7f008c109bf0 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f00740013b0 con 0x7f008c105eb0 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4278631222 0 0) 0x7f008c1a4f40 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 >> v1:172.21.15.119:6789/0 conn(0x7f008c105eb0 legacy=0x7f008c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.285+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 >> v1:172.21.15.114:6789/0 conn(0x7f008c109bf0 legacy=0x7f008c1a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.286+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c1a7340 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.286+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c1a6330 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.286+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c1a6830 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.286+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f007c001b70 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.286+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f007c005d10 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.286+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 13) v1 ==== 45791+0+0 (unknown 1653272359 0 0) 0x7f007c011200 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.287+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c005150 con 0x7f008c101a80 2024-03-24T11:35:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.287+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 352014572 0 0) 0x7f007c046ea0 con 0x7f008c101a80 2024-03-24T11:35:50.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.292+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f007c012b80 con 0x7f008c101a80 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:50.178+0000 7f5e0ef09200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:35:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:35:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:50.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:50.500+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f006c000bc0 con 0x7f007803a060 2024-03-24T11:35:51.395 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[42466]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[42466]: Reconfiguring daemon mgr.y on smithi114 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[30312]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[30312]: Reconfiguring daemon mgr.y on smithi114 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:35:51.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-mon[33714]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:35:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-mon[33714]: Reconfiguring daemon mgr.y on smithi114 2024-03-24T11:35:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:35:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:35:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:51.555 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:51.135+0000 7f5e0ef09200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:35:51.555 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:51.264+0000 7f5e0ef09200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:35:51.877 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:51.596+0000 7f5e0ef09200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:35:51.877 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:51.685+0000 7f5e0ef09200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:35:51.877 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:51.876+0000 7f5e0ef09200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:35:52.304 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:52.058+0000 7f5e0ef09200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:35:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-mon[33714]: from='client.34115 v1:172.21.15.114:0/1097959967' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-mon[33714]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:52.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[42466]: from='client.34115 v1:172.21.15.114:0/1097959967' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:52.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[42466]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:52.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:52.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[30312]: from='client.34115 v1:172.21.15.114:0/1097959967' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:52.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[30312]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:52.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:52.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:52 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:52.597 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:52.340+0000 7f5e0ef09200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:35:52.597 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:52.596+0000 7f5e0ef09200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:35:53.054 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:52.825+0000 7f5e0ef09200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:35:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:53 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:53 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:53 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:53.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:53 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:53.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:53 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:53.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:53 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:53 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:53 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:53 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:35:54.054 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:53.604+0000 7f5e0ef09200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:35:54.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-mon[33714]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3925431789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]: dispatch 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-mon[33714]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]: dispatch 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-mon[33714]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]': finished 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-mon[33714]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:54.161+0000 7f5e0ef09200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:54.255+0000 7f5e0ef09200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:35:54.464 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:35:54 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:35:54.462+0000 7f5e0ef09200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:35:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[30312]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3925431789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]: dispatch 2024-03-24T11:35:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[30312]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]: dispatch 2024-03-24T11:35:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[30312]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]': finished 2024-03-24T11:35:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[30312]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:35:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:35:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[42466]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3925431789' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]: dispatch 2024-03-24T11:35:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[42466]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]: dispatch 2024-03-24T11:35:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[42466]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f7dfcce-3dae-4acc-b2b7-849ea3e22f73"}]': finished 2024-03-24T11:35:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[42466]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:35:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:54 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:35:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:35:55.095+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mgrmap(e 14) v1 ==== 90990+0+0 (unknown 1689017826 0 0) 0x7f007c045740 con 0x7f008c101a80 2024-03-24T11:35:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1609486341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:35:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[30312]: Standby manager daemon x started 2024-03-24T11:35:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:35:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:35:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:35:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:35:55.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1609486341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:35:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[42466]: Standby manager daemon x started 2024-03-24T11:35:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:35:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:35:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:35:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:55 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:35:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1609486341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:35:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:55 smithi119 ceph-mon[33714]: Standby manager daemon x started 2024-03-24T11:35:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:55 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:35:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:55 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:35:55.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:55 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:35:55.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:55 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/1617606577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:35:56.405 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:56 smithi114 ceph-mon[42466]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:56.405 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:56 smithi114 ceph-mon[42466]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-24T11:35:56.405 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:56 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:35:56.406 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:56 smithi114 ceph-mon[30312]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:56.406 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:56 smithi114 ceph-mon[30312]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-24T11:35:56.406 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:56 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:35:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:56 smithi119 ceph-mon[33714]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:56 smithi119 ceph-mon[33714]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-24T11:35:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:56 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:35:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:35:58 smithi114 ceph-mon[30312]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:35:58 smithi114 ceph-mon[42466]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:35:58.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:35:58 smithi119 ceph-mon[33714]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:00 smithi114 ceph-mon[42466]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:00 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:36:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:00 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:00 smithi114 ceph-mon[30312]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:00 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:36:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:00 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:00 smithi119 ceph-mon[33714]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:00 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:36:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:00 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:01.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:01 smithi114 ceph-mon[30312]: Deploying daemon osd.0 on smithi114 2024-03-24T11:36:01.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:01 smithi114 ceph-mon[42466]: Deploying daemon osd.0 on smithi114 2024-03-24T11:36:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:01 smithi119 ceph-mon[33714]: Deploying daemon osd.0 on smithi114 2024-03-24T11:36:02.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:02 smithi114 ceph-mon[30312]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:02.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:02 smithi114 ceph-mon[42466]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:02.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:02 smithi119 ceph-mon[33714]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:04.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:04 smithi114 ceph-mon[30312]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:04.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:04 smithi114 ceph-mon[42466]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:04 smithi119 ceph-mon[33714]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[30312]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:06.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[42466]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:06.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:06.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:06 smithi119 ceph-mon[33714]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:08.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:08 smithi114 ceph-mon[30312]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:08.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:08 smithi114 ceph-mon[42466]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:08.403 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-03-24T11:36:08.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:08.403+0000 7f0088ff9700 1 -- 172.21.15.114:0/1097959967 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1729857151) 0x7f006c000bc0 con 0x7f007803a060 2024-03-24T11:36:08.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:08.405+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 >> v1:172.21.15.114:6800/3103139641 conn(0x7f007803a060 legacy=0x7f007803c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:08.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:08.405+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 >> v1:172.21.15.114:6790/0 conn(0x7f008c101a80 legacy=0x7f008c075930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:08.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:08.405+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 shutdown_connections 2024-03-24T11:36:08.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:08.405+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 >> 172.21.15.114:0/1097959967 conn(0x7f008c0fd020 msgr2=0x7f008c108a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:36:08.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:08.405+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 shutdown_connections 2024-03-24T11:36:08.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:08.405+0000 7f00912d2700 1 -- 172.21.15.114:0/1097959967 wait complete. 2024-03-24T11:36:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:08 smithi119 ceph-mon[33714]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:09.043 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:36:09.045 INFO:tasks.cephadm:Deploying osd.1 on smithi114 with /dev/vg_nvme/lv_3... 2024-03-24T11:36:09.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:36:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:09 smithi119 ceph-mon[33714]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:36:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:09 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:09 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:09.325 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:09.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:09 smithi114 ceph-mon[30312]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:36:09.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:09 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:09.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:09 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:09.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:09 smithi114 ceph-mon[42466]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:36:09.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:09 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:09.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:09 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[42466]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[42466]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[42466]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[42466]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[30312]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[30312]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:36:10.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[30312]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:36:10.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[30312]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:36:10.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:10.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:10.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:10 smithi119 ceph-mon[33714]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:10 smithi119 ceph-mon[33714]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:36:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:10 smithi119 ceph-mon[33714]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:36:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:10 smithi119 ceph-mon[33714]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:36:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:10 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:10 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:10.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:10 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:10.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0[46853]: 2024-03-24T11:36:10.545+0000 7f2fcb8fc700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T11:36:10.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:36:10 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0[46853]: 2024-03-24T11:36:10.547+0000 7f2fc36e7700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:36:11.257 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:36:11.258 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:36:11.258 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-24T11:36:11.258 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-24T11:36:11.258 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012538 s, 836 MB/s 2024-03-24T11:36:11.258 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[30312]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[30312]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[30312]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[42466]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:36:11.292 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[42466]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:36:11.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[42466]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' 2024-03-24T11:36:11.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:11 smithi119 ceph-mon[33714]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:36:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:11 smithi119 ceph-mon[33714]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:36:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:11 smithi119 ceph-mon[33714]: from='osd.0 v1:172.21.15.114:6801/29788729' entity='osd.0' 2024-03-24T11:36:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:12.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[30312]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[30312]: osd.0 v1:172.21.15.114:6801/29788729 boot 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[30312]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[42466]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[42466]: osd.0 v1:172.21.15.114:6801/29788729 boot 2024-03-24T11:36:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[42466]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:36:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:12 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:12 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:36:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:12 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:36:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:12 smithi119 ceph-mon[33714]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:36:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:12 smithi119 ceph-mon[33714]: osd.0 v1:172.21.15.114:6801/29788729 boot 2024-03-24T11:36:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:12 smithi119 ceph-mon[33714]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:36:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:12 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:36:13.115 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:14.472 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:14 smithi114 ceph-mon[30312]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:14.472 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:14 smithi114 ceph-mon[30312]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:36:14.472 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:14 smithi114 ceph-mon[42466]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:14.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:14 smithi114 ceph-mon[42466]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:36:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:14 smithi119 ceph-mon[33714]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:14 smithi119 ceph-mon[33714]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:36:15.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.408+0000 7fdf2f870700 1 -- 172.21.15.114:0/301993431 >> v1:172.21.15.114:6790/0 conn(0x7fdf28105eb0 legacy=0x7fdf28108300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.408+0000 7fdf2f870700 1 -- 172.21.15.114:0/301993431 shutdown_connections 2024-03-24T11:36:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.408+0000 7fdf2f870700 1 -- 172.21.15.114:0/301993431 >> 172.21.15.114:0/301993431 conn(0x7fdf280fd020 msgr2=0x7fdf280ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:36:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.408+0000 7fdf2f870700 1 -- 172.21.15.114:0/301993431 shutdown_connections 2024-03-24T11:36:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.408+0000 7fdf2f870700 1 -- 172.21.15.114:0/301993431 wait complete. 2024-03-24T11:36:15.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.408+0000 7fdf2f870700 1 Processor -- start 2024-03-24T11:36:15.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.408+0000 7fdf2f870700 1 -- start start 2024-03-24T11:36:15.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf2f870700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf281a3f70 con 0x7fdf28109bf0 2024-03-24T11:36:15.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf2f870700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf281a5170 con 0x7fdf28101a80 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf2f870700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf281a6370 con 0x7fdf28105eb0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf2ce0b700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdf28105eb0 0x7fdf2819ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:43208/0 (socket says 172.21.15.114:43208) 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf2ce0b700 1 -- 172.21.15.114:0/328702491 learned_addr learned my addr 172.21.15.114:0/328702491 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272591832 0 0) 0x7fdf281a6370 con 0x7fdf28105eb0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf140035b0 con 0x7fdf28105eb0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440202903 0 0) 0x7fdf281a3f70 con 0x7fdf28109bf0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf281a6370 con 0x7fdf28109bf0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055718541 0 0) 0x7fdf281a5170 con 0x7fdf28101a80 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf281a3f70 con 0x7fdf28101a80 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107089492 0 0) 0x7fdf140035b0 con 0x7fdf28105eb0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf281a5170 con 0x7fdf28105eb0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdf18004540 con 0x7fdf28105eb0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632743746 0 0) 0x7fdf281a6370 con 0x7fdf28109bf0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf140035b0 con 0x7fdf28109bf0 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2285904842 0 0) 0x7fdf281a3f70 con 0x7fdf28101a80 2024-03-24T11:36:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf281a6370 con 0x7fdf28101a80 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdf20001770 con 0x7fdf28109bf0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdf24001360 con 0x7fdf28101a80 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2776190462 0 0) 0x7fdf281a5170 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 >> v1:172.21.15.119:6789/0 conn(0x7fdf28101a80 legacy=0x7fdf28075930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.409+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 >> v1:172.21.15.114:6789/0 conn(0x7fdf28109bf0 legacy=0x7fdf281a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.410+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf281a7570 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.410+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf281a4160 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.410+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf281a4660 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.410+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fdf18003b10 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.410+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdf18004fc0 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.411+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c005150 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.411+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 14) v1 ==== 90990+0+0 (unknown 1689017826 0 0) 0x7fdf18003680 con 0x7fdf28105eb0 2024-03-24T11:36:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.412+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1541+0+0 (unknown 743655088 0 0) 0x7fdf18087320 con 0x7fdf28105eb0 2024-03-24T11:36:15.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.416+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fdf18051ff0 con 0x7fdf28105eb0 2024-03-24T11:36:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:15.623+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf0c000bc0 con 0x7fdf1406e500 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi114 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi114 to 16064M 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi114 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi114 to 16064M 2024-03-24T11:36:15.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:16.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi114 2024-03-24T11:36:16.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:16.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:16.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi114 to 16064M 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:16.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[30312]: from='client.34142 v1:172.21.15.114:0/328702491' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[42466]: from='client.34142 v1:172.21.15.114:0/328702491' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:36:16.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:16 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:16 smithi119 ceph-mon[33714]: from='client.34142 v1:172.21.15.114:0/328702491' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:16 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:36:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:16 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:36:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:16 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:17 smithi114 ceph-mon[30312]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:17 smithi114 ceph-mon[42466]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:18.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:17 smithi119 ceph-mon[33714]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:18.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/884756446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]: dispatch 2024-03-24T11:36:18.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[30312]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]: dispatch 2024-03-24T11:36:18.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[30312]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]': finished 2024-03-24T11:36:18.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[30312]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:36:18.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:18.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/884756446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]: dispatch 2024-03-24T11:36:18.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[42466]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]: dispatch 2024-03-24T11:36:18.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[42466]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]': finished 2024-03-24T11:36:18.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[42466]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:36:18.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:18 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/884756446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]: dispatch 2024-03-24T11:36:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:18 smithi119 ceph-mon[33714]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]: dispatch 2024-03-24T11:36:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:18 smithi119 ceph-mon[33714]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2439ca79-43cf-42d7-95ad-1540a558405e"}]': finished 2024-03-24T11:36:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:18 smithi119 ceph-mon[33714]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:36:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:18 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/279446247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:36:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:19 smithi114 ceph-mon[30312]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/279446247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:36:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:19 smithi114 ceph-mon[42466]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:20.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/279446247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:36:20.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:19 smithi119 ceph-mon[33714]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:22 smithi114 ceph-mon[30312]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:22.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:22 smithi114 ceph-mon[42466]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:22 smithi119 ceph-mon[33714]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:24.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:24 smithi114 ceph-mon[30312]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:24.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:24 smithi114 ceph-mon[42466]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:24 smithi119 ceph-mon[33714]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:25.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:36:25.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:25.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:36:25.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:36:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:26.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:26 smithi114 ceph-mon[30312]: Deploying daemon osd.1 on smithi114 2024-03-24T11:36:26.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:26 smithi114 ceph-mon[30312]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:26.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:26 smithi114 ceph-mon[42466]: Deploying daemon osd.1 on smithi114 2024-03-24T11:36:26.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:26 smithi114 ceph-mon[42466]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:26 smithi119 ceph-mon[33714]: Deploying daemon osd.1 on smithi114 2024-03-24T11:36:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:26 smithi119 ceph-mon[33714]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:28 smithi114 ceph-mon[30312]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:28 smithi114 ceph-mon[42466]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:28.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:28 smithi119 ceph-mon[33714]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:30 smithi114 ceph-mon[30312]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:30 smithi114 ceph-mon[42466]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:30.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:30 smithi119 ceph-mon[33714]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:30.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:30.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:31 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:31.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:31 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:31 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:32.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:32 smithi114 ceph-mon[30312]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:32.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:32 smithi114 ceph-mon[42466]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:32 smithi119 ceph-mon[33714]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:33.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:33.773+0000 7fdf1e7fc700 1 -- 172.21.15.114:0/328702491 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 990800075) 0x7fdf0c000bc0 con 0x7fdf1406e500 2024-03-24T11:36:33.775 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 1 on host 'smithi114' 2024-03-24T11:36:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:33.775+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 >> v1:172.21.15.114:6800/3103139641 conn(0x7fdf1406e500 legacy=0x7fdf140709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:33.775+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 >> v1:172.21.15.114:6790/0 conn(0x7fdf28105eb0 legacy=0x7fdf2819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:33.775+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 shutdown_connections 2024-03-24T11:36:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:33.775+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 >> 172.21.15.114:0/328702491 conn(0x7fdf280fd020 msgr2=0x7fdf28108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:36:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:33.775+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 shutdown_connections 2024-03-24T11:36:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:33.775+0000 7fdf2f870700 1 -- 172.21.15.114:0/328702491 wait complete. 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:33 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:34.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:34.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:34.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:36:34.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:34.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:34.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:33 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:34.419 DEBUG:teuthology.orchestra.run.smithi114:osd.1> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:36:34.421 INFO:tasks.cephadm:Deploying osd.2 on smithi114 with /dev/vg_nvme/lv_2... 2024-03-24T11:36:34.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:36:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:35 smithi119 ceph-mon[33714]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:36:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:35 smithi119 ceph-mon[33714]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:36:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:35 smithi119 ceph-mon[33714]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:36:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:35 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:35.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[30312]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:36:35.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[30312]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:36:35.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[30312]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:36:35.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:35.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[42466]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:36:35.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[42466]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:36:35.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[42466]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:36:35.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:35.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1[52784]: 2024-03-24T11:36:35.402+0000 7f0adbff4700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T11:36:35.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:36:35 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1[52784]: 2024-03-24T11:36:35.405+0000 7f0ad3ddf700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[30312]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[30312]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[30312]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[30312]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[42466]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:36:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[42466]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:36:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[42466]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[42466]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' 2024-03-24T11:36:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:36 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:36 smithi119 ceph-mon[33714]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:36:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:36 smithi119 ceph-mon[33714]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:36:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:36 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:36 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:36 smithi119 ceph-mon[33714]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:36:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:36 smithi119 ceph-mon[33714]: from='osd.1 v1:172.21.15.114:6805/1920152174' entity='osd.1' 2024-03-24T11:36:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:36 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[30312]: osd.1 v1:172.21.15.114:6805/1920152174 boot 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[30312]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[42466]: osd.1 v1:172.21.15.114:6805/1920152174 boot 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[42466]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:36:37.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:37 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:36:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:37 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:36:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:37 smithi119 ceph-mon[33714]: osd.1 v1:172.21.15.114:6805/1920152174 boot 2024-03-24T11:36:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:37 smithi119 ceph-mon[33714]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:36:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:36:37.981 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:36:37.981 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:36:37.981 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-24T11:36:37.982 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-24T11:36:37.982 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150253 s, 698 MB/s 2024-03-24T11:36:37.982 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-24T11:36:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:38 smithi114 ceph-mon[30312]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:38 smithi114 ceph-mon[42466]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:38 smithi119 ceph-mon[33714]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:38.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-03-24T11:36:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:39 smithi114 ceph-mon[30312]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:36:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:39 smithi114 ceph-mon[42466]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:36:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:39 smithi119 ceph-mon[33714]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:36:39.339 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:40 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:40.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:40.470 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:40.470 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:40 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:40.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.944+0000 7fb4a283b700 1 -- 172.21.15.114:0/1244931292 >> v1:172.21.15.114:6789/0 conn(0x7fb49c108ad0 legacy=0x7fb49c10af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:40.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.944+0000 7fb4a283b700 1 -- 172.21.15.114:0/1244931292 shutdown_connections 2024-03-24T11:36:40.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.944+0000 7fb4a283b700 1 -- 172.21.15.114:0/1244931292 >> 172.21.15.114:0/1244931292 conn(0x7fb49c0fbf00 msgr2=0x7fb49c0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:36:40.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.944+0000 7fb4a283b700 1 -- 172.21.15.114:0/1244931292 shutdown_connections 2024-03-24T11:36:40.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.944+0000 7fb4a283b700 1 -- 172.21.15.114:0/1244931292 wait complete. 2024-03-24T11:36:40.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.944+0000 7fb4a283b700 1 Processor -- start 2024-03-24T11:36:40.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.944+0000 7fb4a283b700 1 -- start start 2024-03-24T11:36:40.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4a283b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb49c1a3f60 con 0x7fb49c100960 2024-03-24T11:36:40.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4a283b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb49c1a5160 con 0x7fb49c108ad0 2024-03-24T11:36:40.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4a283b700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb49c1a6360 con 0x7fb49c104d90 2024-03-24T11:36:40.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb49bfff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb49c100960 0x7fb49c106c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58632/0 (socket says 172.21.15.114:58632) 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb49bfff700 1 -- 172.21.15.114:0/3172760865 learned_addr learned my addr 172.21.15.114:0/3172760865 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018635486 0 0) 0x7fb49c1a3f60 con 0x7fb49c100960 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb484005070 con 0x7fb49c100960 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834493686 0 0) 0x7fb49c1a6360 con 0x7fb49c104d90 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c1a3f60 con 0x7fb49c104d90 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507076133 0 0) 0x7fb49c1a5160 con 0x7fb49c108ad0 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c1a6360 con 0x7fb49c108ad0 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1372543683 0 0) 0x7fb484005070 con 0x7fb49c100960 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb49c1a5160 con 0x7fb49c100960 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294650341 0 0) 0x7fb49c1a3f60 con 0x7fb49c104d90 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb484005070 con 0x7fb49c104d90 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb48c0019a0 con 0x7fb49c100960 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.945+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb490001830 con 0x7fb49c104d90 2024-03-24T11:36:40.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925687734 0 0) 0x7fb49c1a6360 con 0x7fb49c108ad0 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb49c1a3f60 con 0x7fb49c108ad0 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb4940017b0 con 0x7fb49c108ad0 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1444267355 0 0) 0x7fb49c1a5160 con 0x7fb49c100960 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 >> v1:172.21.15.114:6790/0 conn(0x7fb49c104d90 legacy=0x7fb49c19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 >> v1:172.21.15.119:6789/0 conn(0x7fb49c108ad0 legacy=0x7fb49c1a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c1a7560 con 0x7fb49c100960 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c1a6550 con 0x7fb49c100960 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c1a6a50 con 0x7fb49c100960 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fb48c003b10 con 0x7fb49c100960 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.946+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb48c005c00 con 0x7fb49c100960 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.947+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb488005150 con 0x7fb49c100960 2024-03-24T11:36:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.947+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 14) v1 ==== 90990+0+0 (unknown 1689017826 0 0) 0x7fb48c003680 con 0x7fb49c100960 2024-03-24T11:36:40.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.948+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1833+0+0 (unknown 4085989735 0 0) 0x7fb48c087030 con 0x7fb49c100960 2024-03-24T11:36:40.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:40.952+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fb48c051be0 con 0x7fb49c100960 2024-03-24T11:36:41.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:41.160+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb488000bc0 con 0x7fb48406e6b0 2024-03-24T11:36:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:41 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi114 2024-03-24T11:36:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:41 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi114 to 8032M 2024-03-24T11:36:41.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:41 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi114 2024-03-24T11:36:41.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:41 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi114 to 8032M 2024-03-24T11:36:41.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:41 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi114 2024-03-24T11:36:41.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:41 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi114 to 8032M 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[30312]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[42466]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:36:42.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:42.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:42 smithi119 ceph-mon[33714]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:42.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:36:42.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:36:42.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:43.156 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:43 smithi114 ceph-mon[30312]: from='client.14274 v1:172.21.15.114:0/3172760865' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:43.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:43 smithi114 ceph-mon[42466]: from='client.14274 v1:172.21.15.114:0/3172760865' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:43.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:43 smithi119 ceph-mon[33714]: from='client.14274 v1:172.21.15.114:0/3172760865' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[30312]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/81041173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]: dispatch 2024-03-24T11:36:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[30312]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]: dispatch 2024-03-24T11:36:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[30312]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]': finished 2024-03-24T11:36:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[30312]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:36:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:36:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[42466]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/81041173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]: dispatch 2024-03-24T11:36:44.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[42466]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]: dispatch 2024-03-24T11:36:44.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[42466]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]': finished 2024-03-24T11:36:44.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[42466]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:36:44.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:44 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:36:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:44 smithi119 ceph-mon[33714]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/81041173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]: dispatch 2024-03-24T11:36:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:44 smithi119 ceph-mon[33714]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]: dispatch 2024-03-24T11:36:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:44 smithi119 ceph-mon[33714]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f1a5efc-c70d-4103-bb5b-d886f0d38540"}]': finished 2024-03-24T11:36:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:44 smithi119 ceph-mon[33714]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:36:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:44 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:36:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1082597378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:36:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1082597378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:36:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1082597378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:36:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:46 smithi114 ceph-mon[30312]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:46 smithi114 ceph-mon[42466]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:46 smithi119 ceph-mon[33714]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:48.114 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:48 smithi114 ceph-mon[30312]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:48.115 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:48 smithi114 ceph-mon[42466]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:48 smithi119 ceph-mon[33714]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:50.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:50 smithi114 ceph-mon[30312]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:50 smithi114 ceph-mon[42466]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:50 smithi119 ceph-mon[33714]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:51.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:36:51.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:51.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:51 smithi114 ceph-mon[30312]: Deploying daemon osd.2 on smithi114 2024-03-24T11:36:51.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:36:51.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:51.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:51 smithi114 ceph-mon[42466]: Deploying daemon osd.2 on smithi114 2024-03-24T11:36:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:36:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:51 smithi119 ceph-mon[33714]: Deploying daemon osd.2 on smithi114 2024-03-24T11:36:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:52 smithi114 ceph-mon[42466]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:52 smithi114 ceph-mon[30312]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:52 smithi119 ceph-mon[33714]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:54 smithi114 ceph-mon[30312]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:54 smithi114 ceph-mon[42466]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:54 smithi119 ceph-mon[33714]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:56.340 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[30312]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:56.340 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:56.340 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:56.340 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:56.341 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[42466]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:56.341 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:56.341 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:56.341 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:56 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:56 smithi119 ceph-mon[33714]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:56 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:56 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:56 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:58 smithi114 ceph-mon[30312]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:58 smithi114 ceph-mon[42466]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:58.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:58 smithi119 ceph-mon[33714]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:59.313 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 2 on host 'smithi114' 2024-03-24T11:36:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:59.312+0000 7fb4997fa700 1 -- 172.21.15.114:0/3172760865 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3744488215) 0x7fb488000bc0 con 0x7fb48406e6b0 2024-03-24T11:36:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:59.314+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 >> v1:172.21.15.114:6800/3103139641 conn(0x7fb48406e6b0 legacy=0x7fb484070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:59.314+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 >> v1:172.21.15.114:6789/0 conn(0x7fb49c100960 legacy=0x7fb49c106c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:36:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:59.314+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 shutdown_connections 2024-03-24T11:36:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:59.315+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 >> 172.21.15.114:0/3172760865 conn(0x7fb49c0fbf00 msgr2=0x7fb49c069dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:36:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:59.315+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 shutdown_connections 2024-03-24T11:36:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:36:59.315+0000 7fb4a283b700 1 -- 172.21.15.114:0/3172760865 wait complete. 2024-03-24T11:36:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:59 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:59 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:59 smithi119 ceph-mon[33714]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:36:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:59 smithi119 ceph-mon[33714]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:59 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:36:59 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.943 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.943 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.943 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[42466]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[42466]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[30312]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[30312]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:36:59 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:36:59.979 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:36:59.981 INFO:tasks.cephadm:Deploying osd.3 on smithi114 with /dev/vg_nvme/lv_1... 2024-03-24T11:36:59.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:37:00.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[30312]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:37:00.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[30312]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:37:00.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[30312]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:00.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:00.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[42466]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:37:00.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[42466]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:37:00.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[42466]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:00.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:00.556 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2[58901]: 2024-03-24T11:37:00.547+0000 7f137e1fa700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T11:37:00.782 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:00 smithi119 ceph-mon[33714]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:37:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:00 smithi119 ceph-mon[33714]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:37:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:00 smithi119 ceph-mon[33714]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:00 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:00.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2[58901]: 2024-03-24T11:37:00.555+0000 7f1379fed700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:37:01.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[30312]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:37:01.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[30312]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:37:01.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:01.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:01.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[30312]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:37:01.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:01.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[42466]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:37:01.859 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[42466]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:37:01.859 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:01.859 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:01.859 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[42466]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:37:01.859 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:01 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:02.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:01 smithi119 ceph-mon[33714]: from='osd.2 v1:172.21.15.114:6809/2539296009' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:37:02.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:01 smithi119 ceph-mon[33714]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:37:02.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:01 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:02.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:01 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:02.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:01 smithi119 ceph-mon[33714]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:37:02.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:01 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:02.956 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:37:02.956 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:37:02.956 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[30312]: osd.2 v1:172.21.15.114:6809/2539296009 boot 2024-03-24T11:37:02.956 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[30312]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:37:02.956 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:02.956 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:37:02.957 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:37:02.957 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[42466]: osd.2 v1:172.21.15.114:6809/2539296009 boot 2024-03-24T11:37:02.957 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[42466]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:37:02.957 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:02 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:02 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:37:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:02 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:37:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:02 smithi119 ceph-mon[33714]: osd.2 v1:172.21.15.114:6809/2539296009 boot 2024-03-24T11:37:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:02 smithi119 ceph-mon[33714]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:37:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:02 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:03.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:03.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:37:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:37:04.014 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-24T11:37:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-24T11:37:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112879 s, 929 MB/s 2024-03-24T11:37:04.015 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 2024-03-24T11:37:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-24T11:37:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:37:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:04.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:04.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:04.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-03-24T11:37:04.844 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:04.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi114 2024-03-24T11:37:04.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi114 to 5354M 2024-03-24T11:37:04.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:04.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[30312]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:37:04.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:04.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi114 2024-03-24T11:37:04.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi114 to 5354M 2024-03-24T11:37:04.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:04.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[42466]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:37:04.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:04 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:05.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:04 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi114 2024-03-24T11:37:05.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:04 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi114 to 5354M 2024-03-24T11:37:05.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:04 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:05.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:04 smithi119 ceph-mon[33714]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:37:05.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:04 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:05.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:05 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:05.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:05 smithi114 ceph-mon[30312]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:37:05.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:05 smithi114 ceph-mon[30312]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:05.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:05 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:05.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:05 smithi114 ceph-mon[42466]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:37:05.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:05 smithi114 ceph-mon[42466]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:05.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63374]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:37:05.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63374]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:05.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63374]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:05.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63374]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:06.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:05 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:06.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:05 smithi119 ceph-mon[33714]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:37:06.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:05 smithi119 ceph-mon[33714]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63377]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:37:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63377]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63377]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:37:05 smithi114 sudo[63377]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.390+0000 7ffa25fad700 1 -- 172.21.15.114:0/672593522 >> v1:172.21.15.114:6789/0 conn(0x7ffa20108ad0 legacy=0x7ffa2010af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.390+0000 7ffa25fad700 1 -- 172.21.15.114:0/672593522 shutdown_connections 2024-03-24T11:37:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.390+0000 7ffa25fad700 1 -- 172.21.15.114:0/672593522 >> 172.21.15.114:0/672593522 conn(0x7ffa200fbf00 msgr2=0x7ffa200fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:37:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.390+0000 7ffa25fad700 1 -- 172.21.15.114:0/672593522 shutdown_connections 2024-03-24T11:37:06.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63400]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:37:06.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63400]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:06.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63400]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:06.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63400]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.391+0000 7ffa25fad700 1 -- 172.21.15.114:0/672593522 wait complete. 2024-03-24T11:37:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.391+0000 7ffa25fad700 1 Processor -- start 2024-03-24T11:37:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.391+0000 7ffa25fad700 1 -- start start 2024-03-24T11:37:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa25fad700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa2010e9e0 con 0x7ffa20108ad0 2024-03-24T11:37:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa25fad700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa20197980 con 0x7ffa20104d90 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa25fad700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa201a4120 con 0x7ffa20100960 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1f7fe700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ffa20100960 0x7ffa20196dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35860/0 (socket says 172.21.15.114:35860) 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1f7fe700 1 -- 172.21.15.114:0/1353529324 learned_addr learned my addr 172.21.15.114:0/1353529324 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577532682 0 0) 0x7ffa201a4120 con 0x7ffa20100960 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa080035b0 con 0x7ffa20100960 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336656554 0 0) 0x7ffa20197980 con 0x7ffa20104d90 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa201a4120 con 0x7ffa20104d90 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3021590108 0 0) 0x7ffa201a4120 con 0x7ffa20104d90 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa20197980 con 0x7ffa20104d90 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ffa100018d0 con 0x7ffa20104d90 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970346746 0 0) 0x7ffa2010e9e0 con 0x7ffa20108ad0 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa201a4120 con 0x7ffa20108ad0 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 193057590 0 0) 0x7ffa080035b0 con 0x7ffa20100960 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa2010e9e0 con 0x7ffa20100960 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ffa14001910 con 0x7ffa20100960 2024-03-24T11:37:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 587899354 0 0) 0x7ffa20197980 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 >> v1:172.21.15.114:6790/0 conn(0x7ffa20100960 legacy=0x7ffa20196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 >> v1:172.21.15.114:6789/0 conn(0x7ffa20108ad0 legacy=0x7ffa201a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.392+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa201a52c0 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.393+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa201a42b0 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.393+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa201a47b0 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.393+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ffa10003b30 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.393+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ffa10004a20 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.394+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0c005150 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.394+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 14) v1 ==== 90990+0+0 (unknown 1689017826 0 0) 0x7ffa10005cb0 con 0x7ffa20104d90 2024-03-24T11:37:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.394+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 2511+0+0 (unknown 3793560836 0 0) 0x7ffa10086010 con 0x7ffa20104d90 2024-03-24T11:37:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.399+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ffa10051820 con 0x7ffa20104d90 2024-03-24T11:37:06.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.588+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7ffa100512d0 con 0x7ffa20104d90 2024-03-24T11:37:06.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:06.613+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa0c000bc0 con 0x7ffa0806e440 2024-03-24T11:37:06.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:06 smithi114 ceph-mon[30312]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:37:06.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:06.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:06 smithi114 ceph-mon[42466]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:37:06.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:06.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63442]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:37:06.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63442]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:06.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63442]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:06.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63442]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:06.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63489]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:37:06.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63489]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:06.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63489]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:06.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63489]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:07.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:06 smithi119 ceph-mon[33714]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:37:07.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:07.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63524]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:37:07.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63524]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:07.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63524]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:07.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 11:37:06 smithi114 sudo[63524]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:07.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63581]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:37:07.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63581]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:07.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63581]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:07.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63581]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63603]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:37:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63603]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63603]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 sudo[63603]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:07.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36535]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:37:07.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:07.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36535]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:07.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36535]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:08.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:37:08.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='client.24199 v1:172.21.15.114:0/1353529324' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:08.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:08.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:08.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:08.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='client.24199 v1:172.21.15.114:0/1353529324' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:08.211 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:08.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63634]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:37:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36538]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:37:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36538]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 sudo[36538]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:37:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='client.24199 v1:172.21.15.114:0/1353529324' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:08.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63634]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:08.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63634]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:08.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63634]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:37:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:37:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:37:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 sudo[63658]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:37:09.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:09.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:09.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:09.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:09.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:09.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:09.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[42466]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:08 smithi114 ceph-mon[30312]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:37:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:37:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:37:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:37:09.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:08 smithi119 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:37:10.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[42466]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:10.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1415537252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34da3c83-66b6-45f3-bf20-6b7fc06e20e4"}]: dispatch 2024-03-24T11:37:10.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1415537252' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34da3c83-66b6-45f3-bf20-6b7fc06e20e4"}]': finished 2024-03-24T11:37:10.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[42466]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:37:10.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:10.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[30312]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:10.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1415537252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34da3c83-66b6-45f3-bf20-6b7fc06e20e4"}]: dispatch 2024-03-24T11:37:10.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1415537252' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34da3c83-66b6-45f3-bf20-6b7fc06e20e4"}]': finished 2024-03-24T11:37:10.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[30312]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:37:10.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:09 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:09 smithi119 ceph-mon[33714]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1415537252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34da3c83-66b6-45f3-bf20-6b7fc06e20e4"}]: dispatch 2024-03-24T11:37:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1415537252' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34da3c83-66b6-45f3-bf20-6b7fc06e20e4"}]': finished 2024-03-24T11:37:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:09 smithi119 ceph-mon[33714]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:37:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:09 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:11.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2247179018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:37:11.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2247179018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:37:11.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2247179018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:37:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:11 smithi114 ceph-mon[30312]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:11.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:11 smithi114 ceph-mon[42466]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:12.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:11 smithi119 ceph-mon[33714]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:14.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:14 smithi114 ceph-mon[30312]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:37:14.441 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:14 smithi114 ceph-mon[42466]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:37:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:14 smithi119 ceph-mon[33714]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:37:16.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:16 smithi114 ceph-mon[30312]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:37:16.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:37:16.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:16.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:16 smithi114 ceph-mon[42466]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:37:16.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:16 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:37:16.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:16 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:16 smithi119 ceph-mon[33714]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:37:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:16 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:37:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:16 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:17.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:17 smithi114 ceph-mon[30312]: Deploying daemon osd.3 on smithi114 2024-03-24T11:37:17.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:17 smithi114 ceph-mon[42466]: Deploying daemon osd.3 on smithi114 2024-03-24T11:37:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:17 smithi119 ceph-mon[33714]: Deploying daemon osd.3 on smithi114 2024-03-24T11:37:18.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:18 smithi114 ceph-mon[30312]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:37:18.419 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:18 smithi114 ceph-mon[42466]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:37:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:18 smithi119 ceph-mon[33714]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:37:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:20 smithi114 ceph-mon[30312]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:37:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:20 smithi114 ceph-mon[42466]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:37:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:20 smithi119 ceph-mon[33714]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:37:22.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[30312]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-24T11:37:22.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:22.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:22.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:22.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[42466]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-24T11:37:22.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:22.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:22.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:22 smithi119 ceph-mon[33714]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-24T11:37:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:24 smithi114 ceph-mon[30312]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-24T11:37:24.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:24 smithi114 ceph-mon[42466]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-24T11:37:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:24 smithi119 ceph-mon[33714]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-24T11:37:25.216 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 3 on host 'smithi114' 2024-03-24T11:37:25.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:25.215+0000 7ffa1cff9700 1 -- 172.21.15.114:0/1353529324 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2200286627) 0x7ffa0c000bc0 con 0x7ffa0806e440 2024-03-24T11:37:25.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:25.217+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 >> v1:172.21.15.114:6800/3103139641 conn(0x7ffa0806e440 legacy=0x7ffa08070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:25.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:25.217+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 >> v1:172.21.15.119:6789/0 conn(0x7ffa20104d90 legacy=0x7ffa2019d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:25.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:25.217+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 shutdown_connections 2024-03-24T11:37:25.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:25.217+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 >> 172.21.15.114:0/1353529324 conn(0x7ffa200fbf00 msgr2=0x7ffa2010bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:37:25.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:25.217+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 shutdown_connections 2024-03-24T11:37:25.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:37:25.217+0000 7ffa25fad700 1 -- 172.21.15.114:0/1353529324 wait complete. 2024-03-24T11:37:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='osd.3 v1:172.21.15.114:6813/3141649135' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:37:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:37:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='osd.3 v1:172.21.15.114:6813/3141649135' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:25 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='osd.3 v1:172.21.15.114:6813/3141649135' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:25.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:25 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:25.857 DEBUG:teuthology.orchestra.run.smithi114:osd.3> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T11:37:25.859 INFO:tasks.cephadm:Deploying osd.4 on smithi119 with /dev/vg_nvme/lv_4... 2024-03-24T11:37:25.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:37:26.101 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:26.478 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:26 smithi119 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:37:26.479 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:26 smithi119 ceph-mon[33714]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:37:26.479 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:26 smithi119 ceph-mon[33714]: from='osd.3 v1:172.21.15.114:6813/3141649135' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:26.479 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:26 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:26.479 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:26 smithi119 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[30312]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[30312]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[30312]: from='osd.3 v1:172.21.15.114:6813/3141649135' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[30312]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[42466]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[42466]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[42466]: from='osd.3 v1:172.21.15.114:6813/3141649135' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:26.711 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-mon[42466]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-24T11:37:26.989 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3[65524]: 2024-03-24T11:37:26.902+0000 7fdb5fe92700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T11:37:26.989 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 11:37:26 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3[65524]: 2024-03-24T11:37:26.906+0000 7fdb57c7d700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:37:27.518 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:27 smithi119 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:37:27.518 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:27 smithi119 ceph-mon[33714]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:37:27.518 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:27 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:27.518 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:27 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:27.518 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:27 smithi119 ceph-mon[33714]: from='osd.3 ' entity='osd.3' 2024-03-24T11:37:27.518 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:27 smithi119 ceph-mon[33714]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[30312]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:37:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[30312]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:37:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[30312]: from='osd.3 ' entity='osd.3' 2024-03-24T11:37:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[30312]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[42466]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-24T11:37:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[42466]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:37:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[42466]: from='osd.3 ' entity='osd.3' 2024-03-24T11:37:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:27 smithi114 ceph-mon[42466]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:37:28.102 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:37:28.102 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:37:28.102 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-24T11:37:28.102 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-24T11:37:28.102 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112654 s, 931 MB/s 2024-03-24T11:37:28.102 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-24T11:37:28.714 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:28 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:37:28.714 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:28 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:37:28.715 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:28.715 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:28 smithi119 ceph-mon[33714]: osd.3 v1:172.21.15.114:6813/3141649135 boot 2024-03-24T11:37:28.715 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:28 smithi119 ceph-mon[33714]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:37:28.715 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:28 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:28.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[30312]: osd.3 v1:172.21.15.114:6813/3141649135 boot 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[30312]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[42466]: osd.3 v1:172.21.15.114:6813/3141649135 boot 2024-03-24T11:37:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[42466]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:37:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:28 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:37:28.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-03-24T11:37:28.976 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:30.229 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi114 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi114 to 4016M 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:30.230 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:30.231 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.231 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:30.231 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:29 smithi119 ceph-mon[33714]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:37:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi114 2024-03-24T11:37:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi114 to 4016M 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[30312]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi114 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:30.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi114 to 4016M 2024-03-24T11:37:30.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:30.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:30.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:30.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:30.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:29 smithi114 ceph-mon[42466]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:37:30.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.774+0000 7f9e33784700 1 -- 172.21.15.119:0/2220028630 >> v1:172.21.15.119:6789/0 conn(0x7f9e2c106000 legacy=0x7f9e2c1083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:30.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.774+0000 7f9e33784700 1 -- 172.21.15.119:0/2220028630 shutdown_connections 2024-03-24T11:37:30.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.774+0000 7f9e33784700 1 -- 172.21.15.119:0/2220028630 >> 172.21.15.119:0/2220028630 conn(0x7f9e2c0fb690 msgr2=0x7f9e2c0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:37:30.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.774+0000 7f9e33784700 1 -- 172.21.15.119:0/2220028630 shutdown_connections 2024-03-24T11:37:30.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.775+0000 7f9e33784700 1 -- 172.21.15.119:0/2220028630 wait complete. 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e33784700 1 Processor -- start 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e33784700 1 -- start start 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e33784700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e2c110cf0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e33784700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e2c197b60 con 0x7f9e2c106000 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e33784700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e2c1a4300 con 0x7f9e2c109ba0 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e31520700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9e2c0ffd50 0x7f9e2c196fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.119:46426/0 (socket says 172.21.15.119:46426) 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e31520700 1 -- 172.21.15.119:0/4169746804 learned_addr learned my addr 172.21.15.119:0/4169746804 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052994292 0 0) 0x7f9e2c110cf0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e180035b0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298116319 0 0) 0x7f9e2c1a4300 con 0x7f9e2c109ba0 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2c110cf0 con 0x7f9e2c109ba0 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169808280 0 0) 0x7f9e2c197b60 con 0x7f9e2c106000 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.776+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2c1a4300 con 0x7f9e2c106000 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3858672712 0 0) 0x7f9e180035b0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e2c197b60 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9e24001cb0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3584881811 0 0) 0x7f9e2c110cf0 con 0x7f9e2c109ba0 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e180035b0 con 0x7f9e2c109ba0 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9e28001830 con 0x7f9e2c109ba0 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4119703334 0 0) 0x7f9e2c1a4300 con 0x7f9e2c106000 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e2c110cf0 con 0x7f9e2c106000 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9e1c001880 con 0x7f9e2c106000 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2344273226 0 0) 0x7f9e2c197b60 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 >> v1:172.21.15.114:6790/0 conn(0x7f9e2c109ba0 legacy=0x7f9e2c1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 >> v1:172.21.15.119:6789/0 conn(0x7f9e2c106000 legacy=0x7f9e2c19d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c1a54a0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c1a44f0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c1a49f0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f9e24002040 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.777+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9e24006090 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.778+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e10005150 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.778+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f9e2401c640 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.779+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(27..27 src has 1..27) v4 ==== 2803+0+0 (unknown 988457707 0 0) 0x7f9e240879b0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.783+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f9e240520c0 con 0x7f9e2c0ffd50 2024-03-24T11:37:30.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:30.991+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e10000bc0 con 0x7f9e1806e5e0 2024-03-24T11:37:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:31 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:31 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:31 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:31.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:31 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:31.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:31 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:31.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:31 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:31.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:31 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:31.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:31 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:31.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:31 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:32.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:32 smithi119 ceph-mon[33714]: from='client.14337 v1:172.21.15.119:0/4169746804' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:32.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:32 smithi119 ceph-mon[33714]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:32 smithi114 ceph-mon[30312]: from='client.14337 v1:172.21.15.119:0/4169746804' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:32 smithi114 ceph-mon[30312]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:32 smithi114 ceph-mon[42466]: from='client.14337 v1:172.21.15.119:0/4169746804' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:32 smithi114 ceph-mon[42466]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[30312]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/783100611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]: dispatch 2024-03-24T11:37:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[30312]: from='client.24226 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]: dispatch 2024-03-24T11:37:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[30312]: from='client.24226 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]': finished 2024-03-24T11:37:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[30312]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:37:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[42466]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/783100611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]: dispatch 2024-03-24T11:37:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[42466]: from='client.24226 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]: dispatch 2024-03-24T11:37:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[42466]: from='client.24226 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]': finished 2024-03-24T11:37:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[42466]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:37:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:34 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:34 smithi119 ceph-mon[33714]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/783100611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]: dispatch 2024-03-24T11:37:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:34 smithi119 ceph-mon[33714]: from='client.24226 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]: dispatch 2024-03-24T11:37:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:34 smithi119 ceph-mon[33714]: from='client.24226 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "305f87ac-0068-49dd-8b7b-24909140de3b"}]': finished 2024-03-24T11:37:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:34 smithi119 ceph-mon[33714]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:37:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:34 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/2640081940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:37:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/2640081940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:37:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/2640081940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:37:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:36 smithi114 ceph-mon[30312]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:36 smithi114 ceph-mon[42466]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:36.518 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:36 smithi119 ceph-mon[33714]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:38.200 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:38 smithi119 ceph-mon[33714]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:38 smithi114 ceph-mon[30312]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:38 smithi114 ceph-mon[42466]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:40.127 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:40 smithi119 ceph-mon[33714]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:40 smithi114 ceph-mon[30312]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:40 smithi114 ceph-mon[42466]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:37:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:41 smithi114 ceph-mon[30312]: Deploying daemon osd.4 on smithi119 2024-03-24T11:37:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:37:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:41 smithi114 ceph-mon[42466]: Deploying daemon osd.4 on smithi119 2024-03-24T11:37:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:37:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:41 smithi119 ceph-mon[33714]: Deploying daemon osd.4 on smithi119 2024-03-24T11:37:42.421 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:42 smithi119 ceph-mon[33714]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:42 smithi114 ceph-mon[30312]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:42 smithi114 ceph-mon[42466]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:44.426 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:44 smithi119 ceph-mon[33714]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:44 smithi114 ceph-mon[30312]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:44 smithi114 ceph-mon[42466]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:46 smithi119 ceph-mon[33714]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:46 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[30312]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[42466]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:46 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:48 smithi114 ceph-mon[30312]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:48 smithi114 ceph-mon[42466]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:48 smithi119 ceph-mon[33714]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:48.953 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 4 on host 'smithi119' 2024-03-24T11:37:48.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:48.951+0000 7f9e227fc700 1 -- 172.21.15.119:0/4169746804 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 292438783) 0x7f9e10000bc0 con 0x7f9e1806e5e0 2024-03-24T11:37:48.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:48.954+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 >> v1:172.21.15.114:6800/3103139641 conn(0x7f9e1806e5e0 legacy=0x7f9e18070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:48.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:48.954+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 >> v1:172.21.15.114:6789/0 conn(0x7f9e2c0ffd50 legacy=0x7f9e2c196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:48.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:48.954+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 shutdown_connections 2024-03-24T11:37:48.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:48.954+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 >> 172.21.15.119:0/4169746804 conn(0x7f9e2c0fb690 msgr2=0x7f9e2c0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:37:48.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:48.954+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 shutdown_connections 2024-03-24T11:37:48.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:48.954+0000 7f9e33784700 1 -- 172.21.15.119:0/4169746804 wait complete. 2024-03-24T11:37:49.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:49 smithi119 ceph-mon[33714]: from='osd.4 v1:172.21.15.119:6800/1375520258' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:37:49.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:49 smithi119 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:37:49.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:49 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:49.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:49 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[30312]: from='osd.4 v1:172.21.15.119:6800/1375520258' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[30312]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[42466]: from='osd.4 v1:172.21.15.119:6800/1375520258' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[42466]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:49 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:49.541 DEBUG:teuthology.orchestra.run.smithi119:osd.4> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T11:37:49.544 INFO:tasks.cephadm:Deploying osd.5 on smithi119 with /dev/vg_nvme/lv_3... 2024-03-24T11:37:49.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:37:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:37:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: from='osd.4 v1:172.21.15.119:6800/1375520258' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:37:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:37:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:37:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:50.305 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T11:37:50.130+0000 7f40bb834700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T11:37:50.305 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 11:37:50 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T11:37:50.136+0000 7f40b7627700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:37:50.340 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: from='osd.4 v1:172.21.15.119:6800/1375520258' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: from='osd.4 v1:172.21.15.119:6800/1375520258' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:50 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:51.424 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:51 smithi119 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:37:51.424 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:51 smithi119 ceph-mon[33714]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:37:51.425 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:51.425 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[30312]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:37:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[30312]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:37:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[42466]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:37:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[42466]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:37:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:52.404 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:52 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:37:52.404 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:52 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:37:52.404 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:52 smithi119 ceph-mon[33714]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:52.404 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:52 smithi119 ceph-mon[33714]: osd.4 v1:172.21.15.119:6800/1375520258 boot 2024-03-24T11:37:52.404 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:52 smithi119 ceph-mon[33714]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:37:52.404 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:52 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:52.405 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:52 smithi119 ceph-mon[33714]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[30312]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[30312]: osd.4 v1:172.21.15.119:6800/1375520258 boot 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[30312]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[30312]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:37:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:37:52.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:37:52.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[42466]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:37:52.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[42466]: osd.4 v1:172.21.15.119:6800/1375520258 boot 2024-03-24T11:37:52.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[42466]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:37:52.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:37:52.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:52 smithi114 ceph-mon[42466]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:37:52.977 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:37:52.977 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:37:52.977 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-24T11:37:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-24T11:37:52.978 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140943 s, 744 MB/s 2024-03-24T11:37:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-24T11:37:53.664 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-03-24T11:37:54.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:54 smithi119 ceph-mon[33714]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:37:54.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:54 smithi119 ceph-mon[33714]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:37:54.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:54 smithi119 ceph-mon[33714]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:37:54.340 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:54 smithi114 ceph-mon[30312]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:37:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:54 smithi114 ceph-mon[30312]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:37:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:54 smithi114 ceph-mon[30312]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:37:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:54 smithi114 ceph-mon[42466]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:37:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:54 smithi114 ceph-mon[42466]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:37:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:54 smithi114 ceph-mon[42466]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:37:55.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi119 2024-03-24T11:37:55.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:55.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi119 to 17088M 2024-03-24T11:37:55.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:37:55.499 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:55 smithi119 ceph-mon[33714]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi119 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi119 to 17088M 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[30312]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi119 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi119 to 17088M 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:55.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:55.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:55.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:55.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:37:55.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:37:55.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:55 smithi114 ceph-mon[42466]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:37:55.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.996+0000 7fbec6562700 1 -- 172.21.15.119:0/3322275945 >> v1:172.21.15.119:6789/0 conn(0x7fbec0104da0 legacy=0x7fbec01071f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:55.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.996+0000 7fbec6562700 1 -- 172.21.15.119:0/3322275945 shutdown_connections 2024-03-24T11:37:55.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.996+0000 7fbec6562700 1 -- 172.21.15.119:0/3322275945 >> 172.21.15.119:0/3322275945 conn(0x7fbec00fbf30 msgr2=0x7fbec00fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:37:55.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.996+0000 7fbec6562700 1 -- 172.21.15.119:0/3322275945 shutdown_connections 2024-03-24T11:37:55.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.996+0000 7fbec6562700 1 -- 172.21.15.119:0/3322275945 wait complete. 2024-03-24T11:37:55.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.997+0000 7fbec6562700 1 Processor -- start 2024-03-24T11:37:55.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.997+0000 7fbec6562700 1 -- start start 2024-03-24T11:37:56.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.997+0000 7fbec6562700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbec01a4090 con 0x7fbec0104da0 2024-03-24T11:37:56.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.997+0000 7fbec6562700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbec01a5290 con 0x7fbec0108ae0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.997+0000 7fbec6562700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbec01a6490 con 0x7fbec0100970 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.997+0000 7fbec4aff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbec0108ae0 0x7fbec01a2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:52250/0 (socket says 172.21.15.119:52250) 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.997+0000 7fbec4aff700 1 -- 172.21.15.119:0/3335678275 learned_addr learned my addr 172.21.15.119:0/3335678275 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041851906 0 0) 0x7fbec01a5290 con 0x7fbec0108ae0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea80035b0 con 0x7fbec0108ae0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005013043 0 0) 0x7fbec01a6490 con 0x7fbec0100970 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbec01a5290 con 0x7fbec0100970 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590067119 0 0) 0x7fbec01a4090 con 0x7fbec0104da0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbec01a6490 con 0x7fbec0104da0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1005032558 0 0) 0x7fbea80035b0 con 0x7fbec0108ae0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbec01a4090 con 0x7fbec0108ae0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fbeb8001ba0 con 0x7fbec0108ae0 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 774344232 0 0) 0x7fbec01a5290 con 0x7fbec0100970 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbea80035b0 con 0x7fbec0100970 2024-03-24T11:37:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fbeb0001830 con 0x7fbec0100970 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2613985533 0 0) 0x7fbec01a6490 con 0x7fbec0104da0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbec01a5290 con 0x7fbec0104da0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fbeb40044b0 con 0x7fbec0104da0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2657946071 0 0) 0x7fbec01a4090 con 0x7fbec0108ae0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 >> v1:172.21.15.114:6790/0 conn(0x7fbec0100970 legacy=0x7fbec0106c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 >> v1:172.21.15.114:6789/0 conn(0x7fbec0104da0 legacy=0x7fbec019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec01a7690 con 0x7fbec0108ae0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec01a5480 con 0x7fbec0108ae0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec01a5980 con 0x7fbec0108ae0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fbeb8003eb0 con 0x7fbec0108ae0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.998+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fbeb8005f50 con 0x7fbec0108ae0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:55.999+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeac005150 con 0x7fbec0108ae0 2024-03-24T11:37:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:56.000+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7fbeb801c520 con 0x7fbec0108ae0 2024-03-24T11:37:56.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:56.000+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(35..35 src has 1..35) v4 ==== 3205+0+0 (unknown 3707946603 0 0) 0x7fbeb8087880 con 0x7fbec0108ae0 2024-03-24T11:37:56.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:56.005+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fbeb8051ed0 con 0x7fbec0108ae0 2024-03-24T11:37:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:37:56.208+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbeac000bc0 con 0x7fbea806e530 2024-03-24T11:37:56.976 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:56 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:56.976 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:56 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:56.976 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:56 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:56.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:56 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:56.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:56 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:56.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:56 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:56 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:37:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:56 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:37:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:56 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:57.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:57 smithi119 ceph-mon[33714]: from='client.24247 v1:172.21.15.119:0/3335678275' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:57.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:57 smithi119 ceph-mon[33714]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-24T11:37:57.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:57 smithi114 ceph-mon[30312]: from='client.24247 v1:172.21.15.119:0/3335678275' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:57.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:57 smithi114 ceph-mon[30312]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-24T11:37:57.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:57 smithi114 ceph-mon[42466]: from='client.24247 v1:172.21.15.119:0/3335678275' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:37:57.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:57 smithi114 ceph-mon[42466]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-24T11:37:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/844474731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]: dispatch 2024-03-24T11:37:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[30312]: from='client.24253 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]: dispatch 2024-03-24T11:37:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[30312]: from='client.24253 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]': finished 2024-03-24T11:37:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[30312]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:37:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:37:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/844474731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]: dispatch 2024-03-24T11:37:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[42466]: from='client.24253 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]: dispatch 2024-03-24T11:37:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[42466]: from='client.24253 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]': finished 2024-03-24T11:37:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[42466]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:37:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:58 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:37:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/844474731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]: dispatch 2024-03-24T11:37:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:58 smithi119 ceph-mon[33714]: from='client.24253 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]: dispatch 2024-03-24T11:37:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:58 smithi119 ceph-mon[33714]: from='client.24253 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7"}]': finished 2024-03-24T11:37:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:58 smithi119 ceph-mon[33714]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:37:59.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:58 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:59 smithi114 ceph-mon[30312]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T11:38:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:37:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/1772876755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:59 smithi114 ceph-mon[42466]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T11:38:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:37:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/1772876755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:00.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:59 smithi119 ceph-mon[33714]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T11:38:00.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:37:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/1772876755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:02.340 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:02 smithi119 ceph-mon[33714]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T11:38:02.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:02 smithi114 ceph-mon[30312]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T11:38:02.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:02 smithi114 ceph-mon[42466]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T11:38:04.350 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:04 smithi119 ceph-mon[33714]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T11:38:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:04 smithi114 ceph-mon[30312]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T11:38:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:04 smithi114 ceph-mon[42466]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T11:38:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:06 smithi119 ceph-mon[33714]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:06.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:38:06.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:06 smithi114 ceph-mon[30312]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:38:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:06 smithi114 ceph-mon[42466]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:38:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:07.372 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:07 smithi119 ceph-mon[33714]: Deploying daemon osd.5 on smithi119 2024-03-24T11:38:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:07 smithi114 ceph-mon[30312]: Deploying daemon osd.5 on smithi119 2024-03-24T11:38:07.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:07 smithi114 ceph-mon[42466]: Deploying daemon osd.5 on smithi119 2024-03-24T11:38:08.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:08 smithi119 ceph-mon[33714]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:08 smithi114 ceph-mon[30312]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:08 smithi114 ceph-mon[42466]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:10.415 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:10 smithi119 ceph-mon[33714]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:10 smithi114 ceph-mon[30312]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:10 smithi114 ceph-mon[42466]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[30312]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:11.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:11 smithi114 ceph-mon[42466]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:12.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:12.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:12.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:12.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:11 smithi119 ceph-mon[33714]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:14.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:14.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:14.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:14.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.363 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: from='osd.5 v1:172.21.15.119:6804/4116082199' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:38:14.363 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:14 smithi119 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:38:14.423 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 5 on host 'smithi119' 2024-03-24T11:38:14.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:14.422+0000 7fbebd7fa700 1 -- 172.21.15.119:0/3335678275 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1299950667) 0x7fbeac000bc0 con 0x7fbea806e530 2024-03-24T11:38:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:14.425+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 >> v1:172.21.15.114:6800/3103139641 conn(0x7fbea806e530 legacy=0x7fbea80709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:14.425+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 >> v1:172.21.15.119:6789/0 conn(0x7fbec0108ae0 legacy=0x7fbec01a2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:14.425+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 shutdown_connections 2024-03-24T11:38:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:14.425+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 >> 172.21.15.119:0/3335678275 conn(0x7fbec00fbf30 msgr2=0x7fbec006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:38:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:14.425+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 shutdown_connections 2024-03-24T11:38:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:14.425+0000 7fbec6562700 1 -- 172.21.15.119:0/3335678275 wait complete. 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: from='osd.5 v1:172.21.15.119:6804/4116082199' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[30312]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:38:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: from='osd.5 v1:172.21.15.119:6804/4116082199' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:38:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:14 smithi114 ceph-mon[42466]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:38:15.091 DEBUG:teuthology.orchestra.run.smithi119:osd.5> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T11:38:15.094 INFO:tasks.cephadm:Deploying osd.6 on smithi119 with /dev/vg_nvme/lv_2... 2024-03-24T11:38:15.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:38:15.619 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:15.619 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:15.620 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:15.620 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:38:15.620 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: from='osd.5 v1:172.21.15.119:6804/4116082199' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:15.620 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:38:15.620 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:15.620 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:15.620 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-mon[33714]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: from='osd.5 v1:172.21.15.119:6804/4116082199' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[30312]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: from='osd.5 v1:172.21.15.119:6804/4116082199' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:15.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:15 smithi114 ceph-mon[42466]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:38:15.911 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:38:16.032 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T11:38:15.631+0000 7fe363efd700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T11:38:16.032 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 11:38:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T11:38:15.636+0000 7fe35fcf0700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:38:16.868 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:16 smithi119 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:38:16.868 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:16 smithi119 ceph-mon[33714]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:38:16.868 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:16 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:16 smithi114 ceph-mon[42466]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:38:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:16 smithi114 ceph-mon[42466]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:38:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:16 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:16 smithi114 ceph-mon[30312]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:38:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:16 smithi114 ceph-mon[30312]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:38:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:16 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:38:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:38:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[30312]: osd.5 v1:172.21.15.119:6804/4116082199 boot 2024-03-24T11:38:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[30312]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:38:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[30312]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[30312]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:38:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:38:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:38:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[42466]: osd.5 v1:172.21.15.119:6804/4116082199 boot 2024-03-24T11:38:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[42466]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:38:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[42466]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:17 smithi114 ceph-mon[42466]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:38:18.001 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:17 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:38:18.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:17 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:38:18.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:17 smithi119 ceph-mon[33714]: osd.5 v1:172.21.15.119:6804/4116082199 boot 2024-03-24T11:38:18.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:17 smithi119 ceph-mon[33714]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:38:18.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:17 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:38:18.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:17 smithi119 ceph-mon[33714]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:18.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:17 smithi119 ceph-mon[33714]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:38:18.744 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:38:18.744 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:38:18.744 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-24T11:38:18.744 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-24T11:38:18.744 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148054 s, 708 MB/s 2024-03-24T11:38:18.744 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-24T11:38:19.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-03-24T11:38:19.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:19 smithi119 ceph-mon[33714]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:38:19.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:19 smithi119 ceph-mon[33714]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:19.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:19 smithi114 ceph-mon[30312]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:38:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:19 smithi114 ceph-mon[30312]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:19 smithi114 ceph-mon[42466]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:38:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:19 smithi114 ceph-mon[42466]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:38:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:38:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi119 2024-03-24T11:38:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:20.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:20.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi119 to 8544M 2024-03-24T11:38:20.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:20.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:20.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:20 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:38:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi119 2024-03-24T11:38:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi119 to 8544M 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi119 2024-03-24T11:38:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi119 to 8544M 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:20.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:20 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:21.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.743+0000 7f99bac41700 1 -- 172.21.15.119:0/1858072590 >> v1:172.21.15.119:6789/0 conn(0x7f99b4109bb0 legacy=0x7f99b410c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:21.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.743+0000 7f99bac41700 1 -- 172.21.15.119:0/1858072590 shutdown_connections 2024-03-24T11:38:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.743+0000 7f99bac41700 1 -- 172.21.15.119:0/1858072590 >> 172.21.15.119:0/1858072590 conn(0x7f99b40fbf00 msgr2=0x7f99b40fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:38:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.743+0000 7f99bac41700 1 -- 172.21.15.119:0/1858072590 shutdown_connections 2024-03-24T11:38:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.743+0000 7f99bac41700 1 -- 172.21.15.119:0/1858072590 wait complete. 2024-03-24T11:38:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.743+0000 7f99bac41700 1 Processor -- start 2024-03-24T11:38:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.743+0000 7f99bac41700 1 -- start start 2024-03-24T11:38:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99bac41700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99b41977b0 con 0x7f99b41060a0 2024-03-24T11:38:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99bac41700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99b41a2fa0 con 0x7f99b4102590 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99bac41700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f99b41a4140 con 0x7f99b4109bb0 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99b89dd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f99b4102590 0x7f99b4196c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54656/0 (socket says 172.21.15.119:54656) 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99b89dd700 1 -- 172.21.15.119:0/2655538777 learned_addr learned my addr 172.21.15.119:0/2655538777 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477785239 0 0) 0x7f99b41a2fa0 con 0x7f99b4102590 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f999c0035b0 con 0x7f99b4102590 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935010559 0 0) 0x7f99b41977b0 con 0x7f99b41060a0 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99b41a2fa0 con 0x7f99b41060a0 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002482763 0 0) 0x7f99b41a4140 con 0x7f99b4109bb0 2024-03-24T11:38:21.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99b41977b0 con 0x7f99b4109bb0 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2136393153 0 0) 0x7f999c0035b0 con 0x7f99b4102590 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99b41a4140 con 0x7f99b4102590 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f99b0004410 con 0x7f99b4102590 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1194609831 0 0) 0x7f99b41a2fa0 con 0x7f99b41060a0 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f999c0035b0 con 0x7f99b41060a0 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f99a40017e0 con 0x7f99b41060a0 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3383139870 0 0) 0x7f99b41977b0 con 0x7f99b4109bb0 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99b41a2fa0 con 0x7f99b4109bb0 2024-03-24T11:38:21.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f99ac0015d0 con 0x7f99b4109bb0 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512079120 0 0) 0x7f99b41a4140 con 0x7f99b4102590 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 >> v1:172.21.15.114:6790/0 conn(0x7f99b4109bb0 legacy=0x7f99b41a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 >> v1:172.21.15.114:6789/0 conn(0x7f99b41060a0 legacy=0x7f99b419d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b41a52e0 con 0x7f99b4102590 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.744+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b41a1ff0 con 0x7f99b4102590 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.745+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f99b00016e0 con 0x7f99b4102590 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.745+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f99b0005fd0 con 0x7f99b4102590 2024-03-24T11:38:21.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.745+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b41a24f0 con 0x7f99b4102590 2024-03-24T11:38:21.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.746+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b410f3f0 con 0x7f99b4102590 2024-03-24T11:38:21.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.746+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f99b0001250 con 0x7f99b4102590 2024-03-24T11:38:21.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.747+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(43..43 src has 1..43) v4 ==== 3497+0+0 (unknown 3007046383 0 0) 0x7f99b0087c80 con 0x7f99b4102590 2024-03-24T11:38:21.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.751+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f99b0052070 con 0x7f99b4102590 2024-03-24T11:38:21.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:21.951+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f99b410ec80 con 0x7f999c06e500 2024-03-24T11:38:21.951 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:21 smithi119 ceph-mon[33714]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:38:21.951 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:21 smithi119 ceph-mon[33714]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:21 smithi114 ceph-mon[30312]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:38:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:21 smithi114 ceph-mon[30312]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:21.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:21 smithi114 ceph-mon[42466]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:38:21.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:21 smithi114 ceph-mon[42466]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:22 smithi119 ceph-mon[33714]: from='client.24274 v1:172.21.15.119:0/2655538777' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:38:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:38:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[30312]: from='client.24274 v1:172.21.15.119:0/2655538777' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[42466]: from='client.24274 v1:172.21.15.119:0/2655538777' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:38:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:23.775 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:23 smithi119 ceph-mon[33714]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-24T11:38:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:23 smithi114 ceph-mon[30312]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-24T11:38:23.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:23 smithi114 ceph-mon[42466]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-24T11:38:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/1833941982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]: dispatch 2024-03-24T11:38:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[30312]: from='client.24280 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]: dispatch 2024-03-24T11:38:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[30312]: from='client.24280 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]': finished 2024-03-24T11:38:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[30312]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:38:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/1833941982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]: dispatch 2024-03-24T11:38:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[42466]: from='client.24280 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]: dispatch 2024-03-24T11:38:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[42466]: from='client.24280 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]': finished 2024-03-24T11:38:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[42466]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:38:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:24 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/1833941982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]: dispatch 2024-03-24T11:38:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:24 smithi119 ceph-mon[33714]: from='client.24280 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]: dispatch 2024-03-24T11:38:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:24 smithi119 ceph-mon[33714]: from='client.24280 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ef919b8-522c-4a53-b548-49a61d64ccce"}]': finished 2024-03-24T11:38:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:24 smithi119 ceph-mon[33714]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:38:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:24 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:25 smithi114 ceph-mon[30312]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:38:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/887399846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:25.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:25 smithi114 ceph-mon[42466]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:38:25.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/887399846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:26.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:25 smithi119 ceph-mon[33714]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:38:26.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/887399846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:28 smithi114 ceph-mon[30312]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T11:38:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:28 smithi114 ceph-mon[42466]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T11:38:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:28 smithi119 ceph-mon[33714]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T11:38:30.220 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:30 smithi119 ceph-mon[33714]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T11:38:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:30 smithi114 ceph-mon[30312]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T11:38:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:30 smithi114 ceph-mon[42466]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T11:38:32.438 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:32 smithi119 ceph-mon[33714]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:32.438 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:32 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:38:32.438 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:32 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:32.438 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:32 smithi119 ceph-mon[33714]: Deploying daemon osd.6 on smithi119 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[30312]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[30312]: Deploying daemon osd.6 on smithi119 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[42466]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:32 smithi114 ceph-mon[42466]: Deploying daemon osd.6 on smithi119 2024-03-24T11:38:34.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:34 smithi119 ceph-mon[33714]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:34.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:34 smithi114 ceph-mon[30312]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:34 smithi114 ceph-mon[42466]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:36 smithi119 ceph-mon[33714]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:36 smithi114 ceph-mon[30312]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:36 smithi114 ceph-mon[42466]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[30312]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:37 smithi114 ceph-mon[42466]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:37 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:37 smithi119 ceph-mon[33714]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:40.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:40 smithi119 ceph-mon[33714]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:40.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:40 smithi119 ceph-mon[33714]: from='osd.6 v1:172.21.15.119:6808/1563073715' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:38:40.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:40 smithi119 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:38:40.315 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 6 on host 'smithi119' 2024-03-24T11:38:40.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:40.314+0000 7f99a9ffb700 1 -- 172.21.15.119:0/2655538777 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2839912343) 0x7f99b410ec80 con 0x7f999c06e500 2024-03-24T11:38:40.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:40.317+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 >> v1:172.21.15.114:6800/3103139641 conn(0x7f999c06e500 legacy=0x7f999c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:40.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:40.317+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 >> v1:172.21.15.119:6789/0 conn(0x7f99b4102590 legacy=0x7f99b4196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:40.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:40.317+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 shutdown_connections 2024-03-24T11:38:40.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:40.317+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 >> 172.21.15.119:0/2655538777 conn(0x7f99b40fbf00 msgr2=0x7f99b40fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:38:40.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:40.317+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 shutdown_connections 2024-03-24T11:38:40.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:40.317+0000 7f99bac41700 1 -- 172.21.15.119:0/2655538777 wait complete. 2024-03-24T11:38:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:40 smithi114 ceph-mon[30312]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:40 smithi114 ceph-mon[30312]: from='osd.6 v1:172.21.15.119:6808/1563073715' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:38:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:40 smithi114 ceph-mon[30312]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:38:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:40 smithi114 ceph-mon[42466]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:40 smithi114 ceph-mon[42466]: from='osd.6 v1:172.21.15.119:6808/1563073715' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:38:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:40 smithi114 ceph-mon[42466]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:38:40.980 DEBUG:teuthology.orchestra.run.smithi119:osd.6> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T11:38:40.982 INFO:tasks.cephadm:Deploying osd.7 on smithi119 with /dev/vg_nvme/lv_1... 2024-03-24T11:38:40.982 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:38:41.257 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='osd.6 v1:172.21.15.119:6808/1563073715' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.258 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:38:41.258 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T11:38:41.103+0000 7f66daf61700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T11:38:41.259 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 11:38:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T11:38:41.113+0000 7f66d6d54700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:38:41.289 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='osd.6 v1:172.21.15.119:6808/1563073715' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[30312]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='osd.6 v1:172.21.15.119:6808/1563073715' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:41 smithi114 ceph-mon[42466]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: osd.6 v1:172.21.15.119:6808/1563073715 boot 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:38:42.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: osd.6 v1:172.21.15.119:6808/1563073715 boot 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:38:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:38:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: osd.6 v1:172.21.15.119:6808/1563073715 boot 2024-03-24T11:38:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:38:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:38:43.267 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:38:43.267 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:38:43.267 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-24T11:38:43.267 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-24T11:38:43.267 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121383 s, 864 MB/s 2024-03-24T11:38:43.267 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-24T11:38:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:43 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:38:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:43 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:38:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:43 smithi119 ceph-mon[33714]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:38:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:43 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:38:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:43 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:38:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:43 smithi114 ceph-mon[30312]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:38:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:43 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:38:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:43 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:38:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:43 smithi114 ceph-mon[42466]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:38:44.443 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:44 smithi119 ceph-mon[33714]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:44.443 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:44 smithi119 ceph-mon[33714]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:38:44.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:44 smithi114 ceph-mon[30312]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:44 smithi114 ceph-mon[30312]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:38:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:44 smithi114 ceph-mon[42466]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:44 smithi114 ceph-mon[42466]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:38:45.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-03-24T11:38:45.611 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:38:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:46 smithi114 ceph-mon[30312]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:46 smithi114 ceph-mon[30312]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:38:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:46 smithi114 ceph-mon[42466]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:46 smithi114 ceph-mon[42466]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:38:46.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:46 smithi119 ceph-mon[33714]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:46.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:46 smithi119 ceph-mon[33714]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:38:47.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:38:47.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:47.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:47.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:47.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:47.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:47.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:47.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:38:47.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.815+0000 7f903bf8b700 1 -- 172.21.15.119:0/1015436512 >> v1:172.21.15.119:6789/0 conn(0x7f9034101fb0 legacy=0x7f90341023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:47.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.815+0000 7f903bf8b700 1 -- 172.21.15.119:0/1015436512 shutdown_connections 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.815+0000 7f903bf8b700 1 -- 172.21.15.119:0/1015436512 >> 172.21.15.119:0/1015436512 conn(0x7f90340fb6e0 msgr2=0x7f90340fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.815+0000 7f903bf8b700 1 -- 172.21.15.119:0/1015436512 shutdown_connections 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f903bf8b700 1 -- 172.21.15.119:0/1015436512 wait complete. 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f903bf8b700 1 Processor -- start 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f903bf8b700 1 -- start start 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f903bf8b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9034100a20 con 0x7f9034104e50 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f903bf8b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90341a2d20 con 0x7f9034101fb0 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f903bf8b700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90341a3f20 con 0x7f9034108b00 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f9039d27700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9034101fb0 0x7f9034199790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58702/0 (socket says 172.21.15.119:58702) 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.816+0000 7f9039d27700 1 -- 172.21.15.119:0/1039985307 learned_addr learned my addr 172.21.15.119:0/1039985307 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596084055 0 0) 0x7f90341a2d20 con 0x7f9034101fb0 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90200035b0 con 0x7f9034101fb0 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454680975 0 0) 0x7f9034100a20 con 0x7f9034104e50 2024-03-24T11:38:47.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90341a2d20 con 0x7f9034104e50 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710484245 0 0) 0x7f90341a3f20 con 0x7f9034108b00 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9034100a20 con 0x7f9034108b00 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4045683350 0 0) 0x7f90200035b0 con 0x7f9034101fb0 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90341a3f20 con 0x7f9034101fb0 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f90300015d0 con 0x7f9034101fb0 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 387186216 0 0) 0x7f90341a2d20 con 0x7f9034104e50 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90200035b0 con 0x7f9034104e50 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f90240014d0 con 0x7f9034104e50 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 721556325 0 0) 0x7f9034100a20 con 0x7f9034108b00 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90341a2d20 con 0x7f9034108b00 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2264338402 0 0) 0x7f90341a3f20 con 0x7f9034101fb0 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 >> v1:172.21.15.114:6790/0 conn(0x7f9034108b00 legacy=0x7f90341a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 >> v1:172.21.15.114:6789/0 conn(0x7f9034104e50 legacy=0x7f903419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90341a5120 con 0x7f9034101fb0 2024-03-24T11:38:47.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90341a1d70 con 0x7f9034101fb0 2024-03-24T11:38:47.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90341a2270 con 0x7f9034101fb0 2024-03-24T11:38:47.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f9030003d60 con 0x7f9034101fb0 2024-03-24T11:38:47.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.817+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9030005190 con 0x7f9034101fb0 2024-03-24T11:38:47.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.818+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9018005150 con 0x7f9034101fb0 2024-03-24T11:38:47.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.818+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f9030003fd0 con 0x7f9034101fb0 2024-03-24T11:38:47.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.819+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(51..51 src has 1..51) v4 ==== 3789+0+0 (unknown 2951099281 0 0) 0x7f9030087d70 con 0x7f9034101fb0 2024-03-24T11:38:47.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:47.824+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f90300520a0 con 0x7f9034101fb0 2024-03-24T11:38:48.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:38:48.024+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9018000bc0 con 0x7f902006e560 2024-03-24T11:38:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:48 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi119 2024-03-24T11:38:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:48 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi119 to 5696M 2024-03-24T11:38:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:48 smithi119 ceph-mon[33714]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:38:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:38:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi119 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi119 to 5696M 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[30312]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi119 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi119 to 5696M 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[42466]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:38:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:38:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:38:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:49 smithi119 ceph-mon[33714]: from='client.24301 v1:172.21.15.119:0/1039985307' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:49 smithi114 ceph-mon[30312]: from='client.24301 v1:172.21.15.119:0/1039985307' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:49 smithi114 ceph-mon[42466]: from='client.24301 v1:172.21.15.119:0/1039985307' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:50 smithi119 ceph-mon[33714]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T11:38:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:50 smithi114 ceph-mon[42466]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T11:38:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:50 smithi114 ceph-mon[30312]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T11:38:51.442 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/4054523221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]: dispatch 2024-03-24T11:38:51.442 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:51 smithi119 ceph-mon[33714]: from='client.24307 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]: dispatch 2024-03-24T11:38:51.442 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:51 smithi119 ceph-mon[33714]: from='client.24307 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]': finished 2024-03-24T11:38:51.442 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:51 smithi119 ceph-mon[33714]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:38:51.442 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:51 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:38:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/4054523221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]: dispatch 2024-03-24T11:38:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[42466]: from='client.24307 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]: dispatch 2024-03-24T11:38:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[42466]: from='client.24307 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]': finished 2024-03-24T11:38:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[42466]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:38:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:38:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/4054523221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]: dispatch 2024-03-24T11:38:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[30312]: from='client.24307 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]: dispatch 2024-03-24T11:38:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[30312]: from='client.24307 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46dea5e4-e52f-43f0-8069-851bdb0d7e62"}]': finished 2024-03-24T11:38:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[30312]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:38:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:51 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:38:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:52 smithi114 ceph-mon[30312]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:38:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/2551968466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:52 smithi114 ceph-mon[42466]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:38:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/2551968466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:52 smithi119 ceph-mon[33714]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:38:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/2551968466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:38:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:54 smithi119 ceph-mon[33714]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:38:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:54 smithi114 ceph-mon[30312]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:38:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:54 smithi114 ceph-mon[42466]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:38:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:56 smithi114 ceph-mon[30312]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:38:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:56 smithi114 ceph-mon[42466]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:38:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:56 smithi119 ceph-mon[33714]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:38:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:58 smithi114 ceph-mon[30312]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:58 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:38:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:58 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:58 smithi114 ceph-mon[42466]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:58 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:38:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:58 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:58 smithi119 ceph-mon[33714]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:38:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:58 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:38:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:58 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:59.392 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:38:59 smithi119 ceph-mon[33714]: Deploying daemon osd.7 on smithi119 2024-03-24T11:38:59.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:38:59 smithi114 ceph-mon[30312]: Deploying daemon osd.7 on smithi119 2024-03-24T11:38:59.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:38:59 smithi114 ceph-mon[42466]: Deploying daemon osd.7 on smithi119 2024-03-24T11:39:00.420 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:00 smithi119 ceph-mon[33714]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:00.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:00 smithi114 ceph-mon[30312]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:00.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:00 smithi114 ceph-mon[42466]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:02.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:02 smithi119 ceph-mon[33714]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:02.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:02 smithi114 ceph-mon[30312]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:02.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:02 smithi114 ceph-mon[42466]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:03 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:03 smithi119 ceph-mon[33714]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[30312]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:03 smithi114 ceph-mon[42466]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:06.232 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:06.232 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.232 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.232 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:06.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:06.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: from='osd.7 v1:172.21.15.119:6812/2022246932' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:39:06.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:06 smithi119 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:39:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: from='osd.7 v1:172.21.15.119:6812/2022246932' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:39:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[30312]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:39:06.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:06.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: from='osd.7 v1:172.21.15.119:6812/2022246932' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:39:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:06 smithi114 ceph-mon[42466]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:39:06.589 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 7 on host 'smithi119' 2024-03-24T11:39:06.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:06.587+0000 7f902affd700 1 -- 172.21.15.119:0/1039985307 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 4115689763) 0x7f9018000bc0 con 0x7f902006e560 2024-03-24T11:39:06.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:06.589+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 >> v1:172.21.15.114:6800/3103139641 conn(0x7f902006e560 legacy=0x7f9020070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:06.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:06.590+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 >> v1:172.21.15.119:6789/0 conn(0x7f9034101fb0 legacy=0x7f9034199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:06.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:06.590+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 shutdown_connections 2024-03-24T11:39:06.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:06.590+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 >> 172.21.15.119:0/1039985307 conn(0x7f90340fb6e0 msgr2=0x7f90341081e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:06.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:06.590+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 shutdown_connections 2024-03-24T11:39:06.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:06.590+0000 7f903bf8b700 1 -- 172.21.15.119:0/1039985307 wait complete. 2024-03-24T11:39:07.387 DEBUG:teuthology.orchestra.run.smithi119:osd.7> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T11:39:07.390 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-24T11:39:07.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:39:07.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: from='osd.7 v1:172.21.15.119:6812/2022246932' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:39:07.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-mon[33714]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:07.644 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:39:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: from='osd.7 v1:172.21.15.119:6812/2022246932' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:39:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:39:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[42466]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: from='osd.7 v1:172.21.15.119:6812/2022246932' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-24T11:39:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:07 smithi114 ceph-mon[30312]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:39:07.890 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T11:39:07.752+0000 7fe385953700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T11:39:07.890 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 11:39:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T11:39:07.759+0000 7fe381746700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:39:08.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[30312]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:39:08.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[30312]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:39:08.751 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[42466]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:39:08.751 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[42466]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:39:08.752 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:08.752 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:08.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:08.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:09.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:08 smithi119 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-24T11:39:09.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:08 smithi119 ceph-mon[33714]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:39:09.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:08 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:09.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:08 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:09.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:08 smithi119 ceph-mon[33714]: osd.7 v1:172.21.15.119:6812/2022246932 boot 2024-03-24T11:39:09.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:08 smithi119 ceph-mon[33714]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:39:09.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:08 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:09.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[30312]: osd.7 v1:172.21.15.119:6812/2022246932 boot 2024-03-24T11:39:09.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[30312]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:39:09.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:09.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[42466]: osd.7 v1:172.21.15.119:6812/2022246932 boot 2024-03-24T11:39:09.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[42466]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:39:09.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:08 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.426+0000 7f109264e700 1 -- 172.21.15.114:0/598627100 >> v1:172.21.15.114:6789/0 conn(0x7f108c108b20 legacy=0x7f108c10afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.426+0000 7f109264e700 1 -- 172.21.15.114:0/598627100 shutdown_connections 2024-03-24T11:39:09.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.426+0000 7f109264e700 1 -- 172.21.15.114:0/598627100 >> 172.21.15.114:0/598627100 conn(0x7f108c0fbf50 msgr2=0x7f108c0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.426+0000 7f109264e700 1 -- 172.21.15.114:0/598627100 shutdown_connections 2024-03-24T11:39:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.426+0000 7f109264e700 1 -- 172.21.15.114:0/598627100 wait complete. 2024-03-24T11:39:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f109264e700 1 Processor -- start 2024-03-24T11:39:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f109264e700 1 -- start start 2024-03-24T11:39:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f109264e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f108c06aec0 con 0x7f108c108b20 2024-03-24T11:39:09.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f109264e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f108c1a2fb0 con 0x7f108c1009b0 2024-03-24T11:39:09.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f109264e700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f108c1a4150 con 0x7f108c104de0 2024-03-24T11:39:09.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f108b7fe700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f108c104de0 0x7f108c19d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52582/0 (socket says 172.21.15.114:52582) 2024-03-24T11:39:09.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f108b7fe700 1 -- 172.21.15.114:0/31295407 learned_addr learned my addr 172.21.15.114:0/31295407 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:39:09.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f1090beb700 1 --1- 172.21.15.114:0/31295407 >> v1:172.21.15.114:6789/0 conn(0x7f108c108b20 0x7f108c1a0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:45424/0 (socket says 172.21.15.114:45424) 2024-03-24T11:39:09.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225917891 0 0) 0x7f108c1a2fb0 con 0x7f108c1009b0 2024-03-24T11:39:09.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10740035b0 con 0x7f108c1009b0 2024-03-24T11:39:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686053660 0 0) 0x7f108c06aec0 con 0x7f108c108b20 2024-03-24T11:39:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f108c1a2fb0 con 0x7f108c108b20 2024-03-24T11:39:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236259309 0 0) 0x7f108c1a4150 con 0x7f108c104de0 2024-03-24T11:39:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f108c06aec0 con 0x7f108c104de0 2024-03-24T11:39:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.427+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 227901877 0 0) 0x7f10740035b0 con 0x7f108c1009b0 2024-03-24T11:39:09.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f108c1a4150 con 0x7f108c1009b0 2024-03-24T11:39:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f10800018b0 con 0x7f108c1009b0 2024-03-24T11:39:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930189710 0 0) 0x7f108c06aec0 con 0x7f108c104de0 2024-03-24T11:39:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10740035b0 con 0x7f108c104de0 2024-03-24T11:39:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 617701931 0 0) 0x7f108c1a2fb0 con 0x7f108c108b20 2024-03-24T11:39:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f108c06aec0 con 0x7f108c108b20 2024-03-24T11:39:09.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f107c001830 con 0x7f108c104de0 2024-03-24T11:39:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f10840046a0 con 0x7f108c108b20 2024-03-24T11:39:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2108595607 0 0) 0x7f108c1a4150 con 0x7f108c1009b0 2024-03-24T11:39:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 >> v1:172.21.15.114:6790/0 conn(0x7f108c104de0 legacy=0x7f108c19d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 >> v1:172.21.15.114:6789/0 conn(0x7f108c108b20 legacy=0x7f108c1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c1a5350 con 0x7f108c1009b0 2024-03-24T11:39:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108c1a4340 con 0x7f108c1009b0 2024-03-24T11:39:09.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f108c1a4840 con 0x7f108c1009b0 2024-03-24T11:39:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f1080003b30 con 0x7f108c1009b0 2024-03-24T11:39:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.428+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f1080004940 con 0x7f108c1009b0 2024-03-24T11:39:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.429+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1078005150 con 0x7f108c1009b0 2024-03-24T11:39:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.429+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f10800036a0 con 0x7f108c1009b0 2024-03-24T11:39:09.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.430+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(55..55 src has 1..55) v4 ==== 4114+0+0 (unknown 1942710602 0 0) 0x7f10800867a0 con 0x7f108c1009b0 2024-03-24T11:39:09.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.434+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f1080050920 con 0x7f108c1009b0 2024-03-24T11:39:09.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.628+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f1078005400 con 0x7f108c1009b0 2024-03-24T11:39:09.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.628+0000 7f10897fa700 1 -- 172.21.15.114:0/31295407 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v55) v1 ==== 74+0+130 (unknown 171817225 0 3355513413) 0x7f1080052850 con 0x7f108c1009b0 2024-03-24T11:39:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:39:09.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.630+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 >> v1:172.21.15.114:6800/3103139641 conn(0x7f107406e740 legacy=0x7f1074070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:09.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.630+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 >> v1:172.21.15.119:6789/0 conn(0x7f108c1009b0 legacy=0x7f108c199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:09.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.630+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 shutdown_connections 2024-03-24T11:39:09.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.630+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 >> 172.21.15.114:0/31295407 conn(0x7f108c0fbf50 msgr2=0x7f108c107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:09.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.630+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 shutdown_connections 2024-03-24T11:39:09.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:09.631+0000 7f109264e700 1 -- 172.21.15.114:0/31295407 wait complete. 2024-03-24T11:39:09.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:09 smithi119 ceph-mon[33714]: purged_snaps scrub starts 2024-03-24T11:39:09.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:09 smithi119 ceph-mon[33714]: purged_snaps scrub ok 2024-03-24T11:39:09.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:09 smithi119 ceph-mon[33714]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:09.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/31295407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:39:09.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:09 smithi119 ceph-mon[33714]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:39:09.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[30312]: purged_snaps scrub starts 2024-03-24T11:39:09.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[30312]: purged_snaps scrub ok 2024-03-24T11:39:09.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[30312]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:09.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/31295407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:39:09.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[30312]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:39:09.945 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[42466]: purged_snaps scrub starts 2024-03-24T11:39:09.945 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[42466]: purged_snaps scrub ok 2024-03-24T11:39:09.946 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[42466]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:09.946 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/31295407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:39:09.946 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:09 smithi114 ceph-mon[42466]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:39:10.354 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711280348,"num_in_osds":8,"osd_in_since":1711280330,"num_remapped_pgs":1} 2024-03-24T11:39:10.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:39:10.592 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: Detected new or changed devices on smithi119 2024-03-24T11:39:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:39:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:11 smithi119 ceph-mon[33714]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: Detected new or changed devices on smithi119 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[30312]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: Detected new or changed devices on smithi119 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:39:11.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:11 smithi114 ceph-mon[42466]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:12.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.337+0000 7f05f7b5a700 1 -- 172.21.15.114:0/3810705129 >> v1:172.21.15.114:6790/0 conn(0x7f05f0107a00 legacy=0x7f05f0107de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:12.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.337+0000 7f05f7b5a700 1 -- 172.21.15.114:0/3810705129 shutdown_connections 2024-03-24T11:39:12.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.337+0000 7f05f7b5a700 1 -- 172.21.15.114:0/3810705129 >> 172.21.15.114:0/3810705129 conn(0x7f05f00fd530 msgr2=0x7f05f00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:12.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.337+0000 7f05f7b5a700 1 -- 172.21.15.114:0/3810705129 shutdown_connections 2024-03-24T11:39:12.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.337+0000 7f05f7b5a700 1 -- 172.21.15.114:0/3810705129 wait complete. 2024-03-24T11:39:12.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.337+0000 7f05f7b5a700 1 Processor -- start 2024-03-24T11:39:12.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.337+0000 7f05f7b5a700 1 -- start start 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05f7b5a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05f01a6860 con 0x7f05f010e480 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05f7b5a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05f01a7a40 con 0x7f05f0107a00 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05f7b5a700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05f01a8c40 con 0x7f05f010a8b0 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05f50f5700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f05f010a8b0 0x7f05f01a1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39412/0 (socket says 172.21.15.114:39412) 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05f50f5700 1 -- 172.21.15.114:0/4237222560 learned_addr learned my addr 172.21.15.114:0/4237222560 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849116349 0 0) 0x7f05f01a8c40 con 0x7f05f010a8b0 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05dc0031f0 con 0x7f05f010a8b0 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164884422 0 0) 0x7f05f01a6860 con 0x7f05f010e480 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f01a8c40 con 0x7f05f010e480 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773910694 0 0) 0x7f05f01a7a40 con 0x7f05f0107a00 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f01a6860 con 0x7f05f0107a00 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1497158261 0 0) 0x7f05dc0031f0 con 0x7f05f010a8b0 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05f01a7a40 con 0x7f05f010a8b0 2024-03-24T11:39:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f05e00018f0 con 0x7f05f010a8b0 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 349851695 0 0) 0x7f05f01a8c40 con 0x7f05f010e480 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05dc0031f0 con 0x7f05f010e480 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f05e80013a0 con 0x7f05f010e480 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1132982399 0 0) 0x7f05f01a6860 con 0x7f05f0107a00 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05f01a8c40 con 0x7f05f0107a00 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f05ec001430 con 0x7f05f0107a00 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4058072211 0 0) 0x7f05f01a7a40 con 0x7f05f010a8b0 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 >> v1:172.21.15.119:6789/0 conn(0x7f05f0107a00 legacy=0x7f05f010da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 >> v1:172.21.15.114:6789/0 conn(0x7f05f010e480 legacy=0x7f05f01a4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f01a9e40 con 0x7f05f010a8b0 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f01a6a50 con 0x7f05f010a8b0 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.338+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f01a6ed0 con 0x7f05f010a8b0 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.339+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f05e0003b30 con 0x7f05f010a8b0 2024-03-24T11:39:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.339+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f05e0005c50 con 0x7f05f010a8b0 2024-03-24T11:39:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.339+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d4005150 con 0x7f05f010a8b0 2024-03-24T11:39:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.340+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f05e00036a0 con 0x7f05f010a8b0 2024-03-24T11:39:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.340+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (unknown 3091298499 0 0) 0x7f05e0087ab0 con 0x7f05f010a8b0 2024-03-24T11:39:12.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.345+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f05e0051cb0 con 0x7f05f010a8b0 2024-03-24T11:39:12.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.537+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05d4005400 con 0x7f05f010a8b0 2024-03-24T11:39:12.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.537+0000 7f05e6ffd700 1 -- 172.21.15.114:0/4237222560 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+11533 (unknown 2626357731 0 2336087866) 0x7f05e0053be0 con 0x7f05f010a8b0 2024-03-24T11:39:12.539 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:39:12.539 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":58,"fsid":"412c996a-e9d2-11ee-95cd-87774f69a715","created":"2024-03-24T11:33:52.997920+0000","modified":"2024-03-24T11:39:11.728968+0000","last_up_change":"2024-03-24T11:39:08.723989+0000","last_in_change":"2024-03-24T11:38:50.789246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:37:03.321298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f7dfcce-3dae-4acc-b2b7-849ea3e22f73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6801","nonce":29788729}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6802","nonce":29788729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6804","nonce":29788729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6803","nonce":29788729}]},"public_addr":"172.21.15.114:6801/29788729","cluster_addr":"172.21.15.114:6802/29788729","heartbeat_back_addr":"172.21.15.114:6804/29788729","heartbeat_front_addr":"172.21.15.114:6803/29788729","state":["exists","up"]},{"osd":1,"uuid":"2439ca79-43cf-42d7-95ad-1540a558405e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6805","nonce":1920152174}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6806","nonce":1920152174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6808","nonce":1920152174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6807","nonce":1920152174}]},"public_addr":"172.21.15.114:6805/1920152174","cluster_addr":"172.21.15.114:6806/1920152174","heartbeat_back_addr":"172.21.15.114:6808/1920152174","heartbeat_front_addr":"172.21.15.114:6807/1920152174","state":["exists","up"]},{"osd":2,"uuid":"1f1a5efc-c70d-4103-bb5b-d886f0d38540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6809","nonce":2539296009}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6810","nonce":2539296009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6812","nonce":2539296009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6811","nonce":2539296009}]},"public_addr":"172.21.15.114:6809/2539296009","cluster_addr":"172.21.15.114:6810/2539296009","heartbeat_back_addr":"172.21.15.114:6812/2539296009","heartbeat_front_addr":"172.21.15.114:6811/2539296009","state":["exists","up"]},{"osd":3,"uuid":"34da3c83-66b6-45f3-bf20-6b7fc06e20e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6813","nonce":3141649135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6814","nonce":3141649135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6816","nonce":3141649135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6815","nonce":3141649135}]},"public_addr":"172.21.15.114:6813/3141649135","cluster_addr":"172.21.15.114:6814/3141649135","heartbeat_back_addr":"172.21.15.114:6816/3141649135","heartbeat_front_addr":"172.21.15.114:6815/3141649135","state":["exists","up"]},{"osd":4,"uuid":"305f87ac-0068-49dd-8b7b-24909140de3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6800","nonce":1375520258}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6801","nonce":1375520258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6803","nonce":1375520258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6802","nonce":1375520258}]},"public_addr":"172.21.15.119:6800/1375520258","cluster_addr":"172.21.15.119:6801/1375520258","heartbeat_back_addr":"172.21.15.119:6803/1375520258","heartbeat_front_addr":"172.21.15.119:6802/1375520258","state":["exists","up"]},{"osd":5,"uuid":"9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6804","nonce":4116082199}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6805","nonce":4116082199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6807","nonce":4116082199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6806","nonce":4116082199}]},"public_addr":"172.21.15.119:6804/4116082199","cluster_addr":"172.21.15.119:6805/4116082199","heartbeat_back_addr":"172.21.15.119:6807/4116082199","heartbeat_front_addr":"172.21.15.119:6806/4116082199","state":["exists","up"]},{"osd":6,"uuid":"5ef919b8-522c-4a53-b548-49a61d64ccce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6808","nonce":1563073715}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6809","nonce":1563073715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6811","nonce":1563073715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6810","nonce":1563073715}]},"public_addr":"172.21.15.119:6808/1563073715","cluster_addr":"172.21.15.119:6809/1563073715","heartbeat_back_addr":"172.21.15.119:6811/1563073715","heartbeat_front_addr":"172.21.15.119:6810/1563073715","state":["exists","up"]},{"osd":7,"uuid":"46dea5e4-e52f-43f0-8069-851bdb0d7e62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6812","nonce":2022246932}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6813","nonce":2022246932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6815","nonce":2022246932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6814","nonce":2022246932}]},"public_addr":"172.21.15.119:6812/2022246932","cluster_addr":"172.21.15.119:6813/2022246932","heartbeat_back_addr":"172.21.15.119:6815/2022246932","heartbeat_front_addr":"172.21.15.119:6814/2022246932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:09.396337+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:34.210168+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:59.825426+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:37:25.558654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:37:49.673911+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:38:14.826924+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:38:40.981951+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:39:06.959944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/3331501023":"2024-03-25T11:34:46.997878+0000","172.21.15.114:0/3312362946":"2024-03-25T11:34:46.997878+0000","172.21.15.114:0/309866353":"2024-03-25T11:34:20.323788+0000","172.21.15.114:6800/1606950246":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/1453553075":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/319492564":"2024-03-25T11:34:20.323788+0000","172.21.15.114:6800/773139778":"2024-03-25T11:34:46.997878+0000","172.21.15.114:0/581628855":"2024-03-25T11:34:46.997878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:39:12.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.540+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 >> v1:172.21.15.114:6800/3103139641 conn(0x7f05dc06e610 legacy=0x7f05dc070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:12.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.540+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 >> v1:172.21.15.114:6790/0 conn(0x7f05f010a8b0 legacy=0x7f05f01a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:12.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.540+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 shutdown_connections 2024-03-24T11:39:12.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.540+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 >> 172.21.15.114:0/4237222560 conn(0x7f05f00fd530 msgr2=0x7f05f0100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:12.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.540+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 shutdown_connections 2024-03-24T11:39:12.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:12.540+0000 7f05f7b5a700 1 -- 172.21.15.114:0/4237222560 wait complete. 2024-03-24T11:39:12.795 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:12 smithi114 ceph-mon[30312]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:39:12.795 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4237222560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:39:12.795 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:12 smithi114 ceph-mon[42466]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:39:12.795 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4237222560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:39:13.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:12 smithi119 ceph-mon[33714]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:39:13.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4237222560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:39:13.108 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:37:03.321298+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T11:39:13.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:39:13.357 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:14.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:13 smithi119 ceph-mon[33714]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:39:14.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:13 smithi119 ceph-mon[33714]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:14.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:13 smithi114 ceph-mon[30312]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:39:14.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:13 smithi114 ceph-mon[30312]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:14.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:13 smithi114 ceph-mon[42466]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:39:14.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:13 smithi114 ceph-mon[42466]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:15.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.011+0000 7f1097113700 1 -- 172.21.15.114:0/1526032771 >> v1:172.21.15.114:6790/0 conn(0x7f109010a940 legacy=0x7f109010cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:15.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.011+0000 7f1097113700 1 -- 172.21.15.114:0/1526032771 shutdown_connections 2024-03-24T11:39:15.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.011+0000 7f1097113700 1 -- 172.21.15.114:0/1526032771 >> 172.21.15.114:0/1526032771 conn(0x7f1090075bf0 msgr2=0x7f1090076000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:15.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.011+0000 7f1097113700 1 -- 172.21.15.114:0/1526032771 shutdown_connections 2024-03-24T11:39:15.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.011+0000 7f1097113700 1 -- 172.21.15.114:0/1526032771 wait complete. 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f1097113700 1 Processor -- start 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f1097113700 1 -- start start 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f1097113700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10901a67f0 con 0x7f10901004c0 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f1097113700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10901a79d0 con 0x7f109010a940 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f1097113700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10901a8bd0 con 0x7f109010e4b0 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f1094eaf700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f10901004c0 0x7f1090106e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46454/0 (socket says 172.21.15.114:46454) 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f1094eaf700 1 -- 172.21.15.114:0/215658289 learned_addr learned my addr 172.21.15.114:0/215658289 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.012+0000 7f10956b0700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f109010e4b0 0x7f10901a4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39430/0 (socket says 172.21.15.114:39430) 2024-03-24T11:39:15.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503040073 0 0) 0x7f10901a67f0 con 0x7f10901004c0 2024-03-24T11:39:15.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10800035b0 con 0x7f10901004c0 2024-03-24T11:39:15.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996388990 0 0) 0x7f10901a8bd0 con 0x7f109010e4b0 2024-03-24T11:39:15.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10901a67f0 con 0x7f109010e4b0 2024-03-24T11:39:15.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279021461 0 0) 0x7f10901a79d0 con 0x7f109010a940 2024-03-24T11:39:15.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10901a8bd0 con 0x7f109010a940 2024-03-24T11:39:15.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 200758567 0 0) 0x7f10800035b0 con 0x7f10901004c0 2024-03-24T11:39:15.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10901a79d0 con 0x7f10901004c0 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f10840018b0 con 0x7f10901004c0 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2622208853 0 0) 0x7f10901a67f0 con 0x7f109010e4b0 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10800035b0 con 0x7f109010e4b0 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f1088001b50 con 0x7f109010e4b0 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857208813 0 0) 0x7f10901a8bd0 con 0x7f109010a940 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10901a67f0 con 0x7f109010a940 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2110819425 0 0) 0x7f10901a79d0 con 0x7f10901004c0 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 >> v1:172.21.15.114:6790/0 conn(0x7f109010e4b0 legacy=0x7f10901a4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 >> v1:172.21.15.119:6789/0 conn(0x7f109010a940 legacy=0x7f10901a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10901a9dd0 con 0x7f10901004c0 2024-03-24T11:39:15.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10901a69e0 con 0x7f10901004c0 2024-03-24T11:39:15.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f1084003b30 con 0x7f10901004c0 2024-03-24T11:39:15.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.013+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f1084005c20 con 0x7f10901004c0 2024-03-24T11:39:15.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.014+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10901a6ee0 con 0x7f10901004c0 2024-03-24T11:39:15.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.014+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1090069c30 con 0x7f10901004c0 2024-03-24T11:39:15.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.015+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f10840036a0 con 0x7f10901004c0 2024-03-24T11:39:15.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.015+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(59..59 src has 1..59) v4 ==== 4081+0+0 (unknown 3684801085 0 0) 0x7f1084087990 con 0x7f10901004c0 2024-03-24T11:39:15.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.020+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f1084051b90 con 0x7f10901004c0 2024-03-24T11:39:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.213+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1090068560 con 0x7f10901004c0 2024-03-24T11:39:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.214+0000 7f108dffb700 1 -- 172.21.15.114:0/215658289 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (unknown 4013917427 0 2170607528) 0x7f1084053ac0 con 0x7f10901004c0 2024-03-24T11:39:15.215 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-24T11:39:15.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.216+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 >> v1:172.21.15.114:6800/3103139641 conn(0x7f108006e540 legacy=0x7f1080070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.216+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 >> v1:172.21.15.114:6789/0 conn(0x7f10901004c0 legacy=0x7f1090106e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.216+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 shutdown_connections 2024-03-24T11:39:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.216+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 >> 172.21.15.114:0/215658289 conn(0x7f1090075bf0 msgr2=0x7f1090111c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.216+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 shutdown_connections 2024-03-24T11:39:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:15.216+0000 7f1097113700 1 -- 172.21.15.114:0/215658289 wait complete. 2024-03-24T11:39:15.794 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi114 2024-03-24T11:39:15.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi114=foo.a' 2024-03-24T11:39:16.043 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:16.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:16 smithi119 ceph-mon[33714]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T11:39:16.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/215658289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:39:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:16 smithi114 ceph-mon[30312]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T11:39:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/215658289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:39:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:16 smithi114 ceph-mon[42466]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T11:39:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/215658289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:39:17.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.633+0000 7fdd51f16700 1 -- 172.21.15.119:0/1828925347 >> v1:172.21.15.119:6789/0 conn(0x7fdd4c101a80 legacy=0x7fdd4c101ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.633+0000 7fdd51f16700 1 -- 172.21.15.119:0/1828925347 shutdown_connections 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.633+0000 7fdd51f16700 1 -- 172.21.15.119:0/1828925347 >> 172.21.15.119:0/1828925347 conn(0x7fdd4c0fd020 msgr2=0x7fdd4c0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.633+0000 7fdd51f16700 1 -- 172.21.15.119:0/1828925347 shutdown_connections 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.633+0000 7fdd51f16700 1 -- 172.21.15.119:0/1828925347 wait complete. 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.633+0000 7fdd51f16700 1 Processor -- start 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd51f16700 1 -- start start 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd51f16700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd4c197990 con 0x7fdd4c105eb0 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd51f16700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd4c1a3180 con 0x7fdd4c101a80 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd51f16700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd4c1a4320 con 0x7fdd4c109bf0 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd4b7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fdd4c101a80 0x7fdd4c196e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:44984/0 (socket says 172.21.15.119:44984) 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd4b7fe700 1 -- 172.21.15.119:0/3772093939 learned_addr learned my addr 172.21.15.119:0/3772093939 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536462358 0 0) 0x7fdd4c1a3180 con 0x7fdd4c101a80 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd340035b0 con 0x7fdd4c101a80 2024-03-24T11:39:17.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311159613 0 0) 0x7fdd340035b0 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd4c1a3180 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdd3c001c80 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708082097 0 0) 0x7fdd4c197990 con 0x7fdd4c105eb0 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd340035b0 con 0x7fdd4c105eb0 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360157143 0 0) 0x7fdd4c1a4320 con 0x7fdd4c109bf0 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd4c197990 con 0x7fdd4c109bf0 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.634+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3611090075 0 0) 0x7fdd4c1a3180 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 >> v1:172.21.15.114:6790/0 conn(0x7fdd4c109bf0 legacy=0x7fdd4c1a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 >> v1:172.21.15.114:6789/0 conn(0x7fdd4c105eb0 legacy=0x7fdd4c19d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c1a54c0 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4c1a21d0 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c1a26d0 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fdd3c001f50 con 0x7fdd4c101a80 2024-03-24T11:39:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdd3c0053a0 con 0x7fdd4c101a80 2024-03-24T11:39:17.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.635+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd38005120 con 0x7fdd4c101a80 2024-03-24T11:39:17.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.636+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7fdd3c0055a0 con 0x7fdd4c101a80 2024-03-24T11:39:17.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.636+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(59..59 src has 1..59) v4 ==== 4081+0+0 (unknown 3684801085 0 0) 0x7fdd3c088090 con 0x7fdd4c101a80 2024-03-24T11:39:17.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.641+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fdd3c052370 con 0x7fdd4c101a80 2024-03-24T11:39:17.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.847+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi114=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd38000b90 con 0x7fdd3406e360 2024-03-24T11:39:17.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.854+0000 7fdd48ff9700 1 -- 172.21.15.119:0/3772093939 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (unknown 0 0 1123153589) 0x7fdd38000b90 con 0x7fdd3406e360 2024-03-24T11:39:17.855 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled rgw.foo.a update... 2024-03-24T11:39:17.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.856+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 >> v1:172.21.15.114:6800/3103139641 conn(0x7fdd3406e360 legacy=0x7fdd34070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:17.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.856+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 >> v1:172.21.15.119:6789/0 conn(0x7fdd4c101a80 legacy=0x7fdd4c196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:17.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.856+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 shutdown_connections 2024-03-24T11:39:17.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.856+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 >> 172.21.15.119:0/3772093939 conn(0x7fdd4c0fd020 msgr2=0x7fdd4c108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:17.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.856+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 shutdown_connections 2024-03-24T11:39:17.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:17.856+0000 7fdd51f16700 1 -- 172.21.15.119:0/3772093939 wait complete. 2024-03-24T11:39:18.414 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:18 smithi119 ceph-mon[33714]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T11:39:18.415 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:18 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:18.415 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:18 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:18.444 DEBUG:teuthology.orchestra.run.smithi114:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-24T11:39:18.446 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi119 2024-03-24T11:39:18.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-24T11:39:18.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:18 smithi114 ceph-mon[30312]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T11:39:18.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:18.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:18 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:18.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:18 smithi114 ceph-mon[42466]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T11:39:18.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:18 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:18.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:18 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:18.678 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:19.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='client.24346 v1:172.21.15.119:0/3772093939' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi114=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='client.24346 v1:172.21.15.119:0/3772093939' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi114=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:19.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:19.307 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:19.307 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:39:19.307 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:39:19.307 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:19.307 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:19 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='client.24346 v1:172.21.15.119:0/3772093939' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi114=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:19.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:19 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:20.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.289+0000 7f67b6f2a700 1 -- 172.21.15.119:0/76352159 >> v1:172.21.15.119:6789/0 conn(0x7f67b0109bf0 legacy=0x7f67b010c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:20.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.289+0000 7f67b6f2a700 1 -- 172.21.15.119:0/76352159 shutdown_connections 2024-03-24T11:39:20.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.289+0000 7f67b6f2a700 1 -- 172.21.15.119:0/76352159 >> 172.21.15.119:0/76352159 conn(0x7f67b00fd020 msgr2=0x7f67b00ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:20.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.289+0000 7f67b6f2a700 1 -- 172.21.15.119:0/76352159 shutdown_connections 2024-03-24T11:39:20.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.289+0000 7f67b6f2a700 1 -- 172.21.15.119:0/76352159 wait complete. 2024-03-24T11:39:20.291 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:20 smithi119 ceph-mon[33714]: Deploying daemon rgw.foo.a on smithi114 2024-03-24T11:39:20.291 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:20 smithi119 ceph-mon[33714]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.290+0000 7f67b6f2a700 1 Processor -- start 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.290+0000 7f67b6f2a700 1 -- start start 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.290+0000 7f67b6f2a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f67b0110d00 con 0x7f67b0105eb0 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.290+0000 7f67b6f2a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f67b00ff240 con 0x7f67b0101a80 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.290+0000 7f67b6f2a700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f67b00ff3d0 con 0x7f67b0109bf0 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.290+0000 7f67b4cc6700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f67b0101a80 0x7f67b00fe910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34524/0 (socket says 172.21.15.119:34524) 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.290+0000 7f67b4cc6700 1 -- 172.21.15.119:0/2925180620 learned_addr learned my addr 172.21.15.119:0/2925180620 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549634689 0 0) 0x7f67b0110d00 con 0x7f67b0105eb0 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a00035b0 con 0x7f67b0105eb0 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563620831 0 0) 0x7f67b00ff240 con 0x7f67b0101a80 2024-03-24T11:39:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b0110d00 con 0x7f67b0101a80 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398198940 0 0) 0x7f67b00ff3d0 con 0x7f67b0109bf0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b00ff240 con 0x7f67b0109bf0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3674621415 0 0) 0x7f67a00035b0 con 0x7f67b0105eb0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b00ff3d0 con 0x7f67b0105eb0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f679c001830 con 0x7f67b0105eb0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2391299774 0 0) 0x7f67b00ff240 con 0x7f67b0109bf0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67a00035b0 con 0x7f67b0109bf0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477197136 0 0) 0x7f67b0110d00 con 0x7f67b0101a80 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b00ff240 con 0x7f67b0101a80 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f67a8004860 con 0x7f67b0109bf0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f67a4001760 con 0x7f67b0101a80 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 661110672 0 0) 0x7f67b00ff3d0 con 0x7f67b0105eb0 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 >> v1:172.21.15.114:6790/0 conn(0x7f67b0109bf0 legacy=0x7f67b0199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 >> v1:172.21.15.119:6789/0 conn(0x7f67b0101a80 legacy=0x7f67b00fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00ff560 con 0x7f67b0105eb0 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b01a8c50 con 0x7f67b0105eb0 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f679c003b30 con 0x7f67b0105eb0 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f679c004a00 con 0x7f67b0105eb0 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.291+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b01a9150 con 0x7f67b0105eb0 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.292+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b0068560 con 0x7f67b0105eb0 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.293+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f679c0036a0 con 0x7f67b0105eb0 2024-03-24T11:39:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.293+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(59..59 src has 1..59) v4 ==== 4081+0+0 (unknown 3684801085 0 0) 0x7f679c086850 con 0x7f67b0105eb0 2024-03-24T11:39:20.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.298+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f679c0509f0 con 0x7f67b0105eb0 2024-03-24T11:39:20.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:20 smithi114 ceph-mon[30312]: Deploying daemon rgw.foo.a on smithi114 2024-03-24T11:39:20.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:20 smithi114 ceph-mon[30312]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:39:20.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:20 smithi114 ceph-mon[42466]: Deploying daemon rgw.foo.a on smithi114 2024-03-24T11:39:20.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:20 smithi114 ceph-mon[42466]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:39:20.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:20.491+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f67b010ec80 con 0x7f67b0105eb0 2024-03-24T11:39:20.697 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 11:39:20 smithi114 systemd[1]: Starting Ceph rgw.foo.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:39:20.973 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 11:39:20 smithi114 podman[70484]: 2024-03-24 11:39:20.599440543 +0000 UTC m=+0.017380457 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:39:20.973 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 11:39:20 smithi114 podman[70484]: 2024-03-24 11:39:20.728621932 +0000 UTC m=+0.146561843 container create 669e0c06dcb5abbfc02a48c1ee9bdbee3ae1053003f29a3cead2c68da528bc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T11:39:20.973 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 11:39:20 smithi114 podman[70484]: 2024-03-24 11:39:20.972921345 +0000 UTC m=+0.390861262 container init 669e0c06dcb5abbfc02a48c1ee9bdbee3ae1053003f29a3cead2c68da528bc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T11:39:21.239 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 11:39:20 smithi114 podman[70484]: 2024-03-24 11:39:20.978856686 +0000 UTC m=+0.396796602 container start 669e0c06dcb5abbfc02a48c1ee9bdbee3ae1053003f29a3cead2c68da528bc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T11:39:21.239 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 11:39:21 smithi114 bash[70484]: 669e0c06dcb5abbfc02a48c1ee9bdbee3ae1053003f29a3cead2c68da528bc89 2024-03-24T11:39:21.239 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 11:39:21 smithi114 systemd[1]: Started Ceph rgw.foo.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:39:21.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:21.257+0000 7f67adffb700 1 -- 172.21.15.119:0/2925180620 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v60) v1 ==== 160+0+0 (unknown 701681216 0 0) 0x7f679c052920 con 0x7f67b0105eb0 2024-03-24T11:39:21.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:21.262+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 >> v1:172.21.15.114:6800/3103139641 conn(0x7f67a006e5b0 legacy=0x7f67a0070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:21.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:21.262+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 >> v1:172.21.15.114:6789/0 conn(0x7f67b0105eb0 legacy=0x7f67b0199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:21.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:21.262+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 shutdown_connections 2024-03-24T11:39:21.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:21.262+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 >> 172.21.15.119:0/2925180620 conn(0x7f67b00fd020 msgr2=0x7f67b010c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:21.264+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 shutdown_connections 2024-03-24T11:39:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:21.264+0000 7f67b6f2a700 1 -- 172.21.15.119:0/2925180620 wait complete. 2024-03-24T11:39:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'datapool' created 2024-03-24T11:39:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/2925180620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:39:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/2925180620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:39:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/2925180620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:39:21.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- rbd pool init datapool 2024-03-24T11:39:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/2925180620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1876490541' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/2925180620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1876490541' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:22.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:22 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T11:39:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/2925180620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:39:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:39:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1876490541' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:39:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-03-24T11:39:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:22 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:23.343 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1876490541' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:39:23.344 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:23 smithi119 ceph-mon[33714]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:39:23.344 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:23.344 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:23.344 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:23.344 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:23 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1876490541' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[42466]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1876490541' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[30312]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:23.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:23 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: Checking dashboard <-> RGW credentials 2024-03-24T11:39:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: pgmap v189: 36 pgs: 17 creating+peering, 2 active+clean, 17 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:39:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/1376217807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:39:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: from='client.24403 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:39:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:24 smithi119 ceph-mon[33714]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: Checking dashboard <-> RGW credentials 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: pgmap v189: 36 pgs: 17 creating+peering, 2 active+clean, 17 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/1376217807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: from='client.24403 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:39:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[42466]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: Checking dashboard <-> RGW credentials 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: pgmap v189: 36 pgs: 17 creating+peering, 2 active+clean, 17 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/1376217807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: from='client.24403 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:39:24.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:24 smithi114 ceph-mon[30312]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[30312]: from='client.24403 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[30312]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[30312]: pgmap v192: 68 pgs: 24 creating+peering, 15 active+clean, 29 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[42466]: from='client.24403 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:39:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[42466]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:39:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:25 smithi114 ceph-mon[42466]: pgmap v192: 68 pgs: 24 creating+peering, 15 active+clean, 29 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:39:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:39:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:25 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:39:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:25 smithi119 ceph-mon[33714]: from='client.24403 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:39:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:25 smithi119 ceph-mon[33714]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:39:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:25 smithi119 ceph-mon[33714]: pgmap v192: 68 pgs: 24 creating+peering, 15 active+clean, 29 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:39:26.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:26 smithi119 ceph-mon[33714]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:39:26.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:26 smithi119 ceph-mon[33714]: Cluster is now healthy 2024-03-24T11:39:26.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:26 smithi119 ceph-mon[33714]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:39:26.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:26 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[30312]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[30312]: Cluster is now healthy 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[30312]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[42466]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[42466]: Cluster is now healthy 2024-03-24T11:39:26.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[42466]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:39:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:26 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:39:27.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.119 --placement '1;smithi119=iscsi.a' 2024-03-24T11:39:27.391 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:27.512 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:39:27.512 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:27 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:39:27.512 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:27 smithi119 ceph-mon[33714]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:39:27.512 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:27 smithi119 ceph-mon[33714]: pgmap v195: 100 pgs: 19 creating+peering, 53 active+clean, 28 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[30312]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[30312]: pgmap v195: 100 pgs: 19 creating+peering, 53 active+clean, 28 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[42466]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:39:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:27 smithi114 ceph-mon[42466]: pgmap v195: 100 pgs: 19 creating+peering, 53 active+clean, 28 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T11:39:28.730 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:28 smithi119 ceph-mon[33714]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:39:28.730 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.730 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:28 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.731 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[30312]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[42466]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:39:29.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.088+0000 7fdb01380700 1 -- 172.21.15.119:0/795073211 >> v1:172.21.15.119:6789/0 conn(0x7fdafc100940 legacy=0x7fdafc100d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:29.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.088+0000 7fdb01380700 1 -- 172.21.15.119:0/795073211 shutdown_connections 2024-03-24T11:39:29.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.088+0000 7fdb01380700 1 -- 172.21.15.119:0/795073211 >> 172.21.15.119:0/795073211 conn(0x7fdafc0fbf00 msgr2=0x7fdafc0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:29.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.088+0000 7fdb01380700 1 -- 172.21.15.119:0/795073211 shutdown_connections 2024-03-24T11:39:29.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdb01380700 1 -- 172.21.15.119:0/795073211 wait complete. 2024-03-24T11:39:29.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdb01380700 1 Processor -- start 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdb01380700 1 -- start start 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdb01380700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdafc06aca0 con 0x7fdafc100940 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdb01380700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdafc06ae90 con 0x7fdafc108ab0 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdb01380700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdafc1a61b0 con 0x7fdafc104d70 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdafb7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fdafc108ab0 0x7fdafc1a2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48772/0 (socket says 172.21.15.119:48772) 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.089+0000 7fdafb7fe700 1 -- 172.21.15.119:0/2360853853 learned_addr learned my addr 172.21.15.119:0/2360853853 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082267128 0 0) 0x7fdafc06ae90 con 0x7fdafc108ab0 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae80035b0 con 0x7fdafc108ab0 2024-03-24T11:39:29.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872052149 0 0) 0x7fdafc06aca0 con 0x7fdafc100940 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdafc06ae90 con 0x7fdafc100940 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942928524 0 0) 0x7fdafc1a61b0 con 0x7fdafc104d70 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdafc06aca0 con 0x7fdafc104d70 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4014958247 0 0) 0x7fdae80035b0 con 0x7fdafc108ab0 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdafc1a61b0 con 0x7fdafc108ab0 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdaf4004390 con 0x7fdafc108ab0 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3437187076 0 0) 0x7fdafc06aca0 con 0x7fdafc104d70 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdae80035b0 con 0x7fdafc104d70 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdae40014d0 con 0x7fdafc104d70 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712364894 0 0) 0x7fdafc06ae90 con 0x7fdafc100940 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdafc06aca0 con 0x7fdafc100940 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2257562776 0 0) 0x7fdafc1a61b0 con 0x7fdafc108ab0 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 >> v1:172.21.15.114:6790/0 conn(0x7fdafc104d70 legacy=0x7fdafc19f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 >> v1:172.21.15.114:6789/0 conn(0x7fdafc100940 legacy=0x7fdafc19bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc1a7350 con 0x7fdafc108ab0 2024-03-24T11:39:29.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc1a63a0 con 0x7fdafc108ab0 2024-03-24T11:39:29.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.090+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc1a68a0 con 0x7fdafc108ab0 2024-03-24T11:39:29.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.091+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fdaf40016e0 con 0x7fdafc108ab0 2024-03-24T11:39:29.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.091+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdaf4005f30 con 0x7fdafc108ab0 2024-03-24T11:39:29.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.091+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdadc005150 con 0x7fdafc108ab0 2024-03-24T11:39:29.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.092+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7fdaf401c440 con 0x7fdafc108ab0 2024-03-24T11:39:29.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.092+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(67..67 src has 1..67) v4 ==== 5926+0+0 (unknown 3694954280 0 0) 0x7fdaf40883c0 con 0x7fdafc108ab0 2024-03-24T11:39:29.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.097+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fdaf4051ff0 con 0x7fdafc108ab0 2024-03-24T11:39:29.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.305+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.119", "placement": "1;smithi119=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fdadc000c90 con 0x7fdae806e560 2024-03-24T11:39:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.313+0000 7fdaf8ff9700 1 -- 172.21.15.119:0/2360853853 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (unknown 0 0 803663096) 0x7fdadc000c90 con 0x7fdae806e560 2024-03-24T11:39:29.314 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled iscsi.datapool update... 2024-03-24T11:39:29.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.315+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 >> v1:172.21.15.114:6800/3103139641 conn(0x7fdae806e560 legacy=0x7fdae8070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:29.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.315+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 >> v1:172.21.15.119:6789/0 conn(0x7fdafc108ab0 legacy=0x7fdafc1a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:29.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.315+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 shutdown_connections 2024-03-24T11:39:29.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.315+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 >> 172.21.15.119:0/2360853853 conn(0x7fdafc0fbf00 msgr2=0x7fdafc10b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:29.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.315+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 shutdown_connections 2024-03-24T11:39:29.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:29.315+0000 7fdb01380700 1 -- 172.21.15.119:0/2360853853 wait complete. 2024-03-24T11:39:29.640 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:39:29.640 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:39:29.640 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:39:29.641 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.641 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.641 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.641 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: pgmap v198: 132 pgs: 7 creating+activating, 6 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T11:39:29.641 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:29 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: pgmap v198: 132 pgs: 7 creating+activating, 6 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T11:39:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/106967721' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: pgmap v198: 132 pgs: 7 creating+activating, 6 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-24T11:39:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:29 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:29.962 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-24T11:39:29.963 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:39:29.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T11:39:29.993 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:39:29.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T11:39:30.024 DEBUG:teuthology.orchestra.run.smithi119:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@iscsi.iscsi.a.service 2024-03-24T11:39:30.067 INFO:tasks.cephadm:Adding prometheus.a on smithi119 2024-03-24T11:39:30.067 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi119=a' 2024-03-24T11:39:30.536 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='client.24409 v1:172.21.15.119:0/2360853853' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.119", "placement": "1;smithi119=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:30.536 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: Saving service iscsi.datapool spec with placement smithi119=iscsi.a;count:1 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:30.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:30 smithi119 ceph-mon[33714]: Deploying daemon iscsi.iscsi.a on smithi119 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='client.24409 v1:172.21.15.119:0/2360853853' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.119", "placement": "1;smithi119=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: Saving service iscsi.datapool spec with placement smithi119=iscsi.a;count:1 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[30312]: Deploying daemon iscsi.iscsi.a on smithi119 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='client.24409 v1:172.21.15.119:0/2360853853' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.119", "placement": "1;smithi119=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: Saving service iscsi.datapool spec with placement smithi119=iscsi.a;count:1 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1056840841' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='client.34361 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:39:30.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:39:30.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:30.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:30 smithi114 ceph-mon[42466]: Deploying daemon iscsi.iscsi.a on smithi119 2024-03-24T11:39:30.829 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:31.509 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:31 smithi119 ceph-mon[33714]: pgmap v200: 132 pgs: 7 creating+activating, 6 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T11:39:31.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:31 smithi114 ceph-mon[30312]: pgmap v200: 132 pgs: 7 creating+activating, 6 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T11:39:31.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:31 smithi114 ceph-mon[42466]: pgmap v200: 132 pgs: 7 creating+activating, 6 creating+peering, 102 active+clean, 17 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T11:39:33.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:32 smithi119 podman[62377]: 2024-03-24 11:39:32.935957523 +0000 UTC m=+0.332885995 container create 9f94706c27741e452ce286ae70f7732a5403286b2c3e5471ef53680b7a14abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:39:33.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 podman[62377]: 2024-03-24 11:39:33.279963973 +0000 UTC m=+0.676892451 container init 9f94706c27741e452ce286ae70f7732a5403286b2c3e5471ef53680b7a14abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T11:39:33.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 podman[62377]: 2024-03-24 11:39:33.28519674 +0000 UTC m=+0.682125213 container start 9f94706c27741e452ce286ae70f7732a5403286b2c3e5471ef53680b7a14abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T11:39:33.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 bash[62377]: 9f94706c27741e452ce286ae70f7732a5403286b2c3e5471ef53680b7a14abef 2024-03-24T11:39:33.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 systemd[1]: Started Ceph iscsi.iscsi.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.676+0000 7f9451311700 1 -- 172.21.15.119:0/1137311249 >> v1:172.21.15.119:6789/0 conn(0x7f944c102540 legacy=0x7f944c104930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.676+0000 7f9451311700 1 -- 172.21.15.119:0/1137311249 shutdown_connections 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.676+0000 7f9451311700 1 -- 172.21.15.119:0/1137311249 >> 172.21.15.119:0/1137311249 conn(0x7f944c0fbf70 msgr2=0x7f944c0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.676+0000 7f9451311700 1 -- 172.21.15.119:0/1137311249 shutdown_connections 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.676+0000 7f9451311700 1 -- 172.21.15.119:0/1137311249 wait complete. 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.677+0000 7f9451311700 1 Processor -- start 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.677+0000 7f9451311700 1 -- start start 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.677+0000 7f9451311700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f944c1979d0 con 0x7f944c109bf0 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.677+0000 7f9451311700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f944c1a31c0 con 0x7f944c102540 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.677+0000 7f9451311700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f944c1a4360 con 0x7f944c106050 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f944a7fc700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f944c106050 0x7f944c19d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.119:43886/0 (socket says 172.21.15.119:43886) 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f944b7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f944c109bf0 0x7f944c1a0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.119:40674/0 (socket says 172.21.15.119:40674) 2024-03-24T11:39:33.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f944a7fc700 1 -- 172.21.15.119:0/2487228771 learned_addr learned my addr 172.21.15.119:0/2487228771 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201819312 0 0) 0x7f944c1979d0 con 0x7f944c109bf0 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94340037f0 con 0x7f944c109bf0 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358569517 0 0) 0x7f944c1a4360 con 0x7f944c106050 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f944c1979d0 con 0x7f944c106050 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129510658 0 0) 0x7f944c1a31c0 con 0x7f944c102540 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f944c1a4360 con 0x7f944c102540 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 529836365 0 0) 0x7f94340037f0 con 0x7f944c109bf0 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f944c1a31c0 con 0x7f944c109bf0 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9444001bb0 con 0x7f944c109bf0 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3910801002 0 0) 0x7f944c1979d0 con 0x7f944c106050 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94340037f0 con 0x7f944c106050 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1410167924 0 0) 0x7f944c1a4360 con 0x7f944c102540 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f944c1979d0 con 0x7f944c102540 2024-03-24T11:39:33.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f94400018f0 con 0x7f944c106050 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1061028528 0 0) 0x7f944c1a31c0 con 0x7f944c109bf0 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 >> v1:172.21.15.114:6790/0 conn(0x7f944c106050 legacy=0x7f944c19d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 >> v1:172.21.15.119:6789/0 conn(0x7f944c102540 legacy=0x7f944c196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c1a5500 con 0x7f944c109bf0 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c1a2210 con 0x7f944c109bf0 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.678+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c1a2710 con 0x7f944c109bf0 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.679+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f94440020c0 con 0x7f944c109bf0 2024-03-24T11:39:33.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.679+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9444005dd0 con 0x7f944c109bf0 2024-03-24T11:39:33.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.680+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9438005150 con 0x7f944c109bf0 2024-03-24T11:39:33.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.680+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 15) v1 ==== 91046+0+0 (unknown 1600015731 0 0) 0x7f944401c2e0 con 0x7f944c109bf0 2024-03-24T11:39:33.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.680+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (unknown 1724832888 0 0) 0x7f9444088310 con 0x7f944c109bf0 2024-03-24T11:39:33.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.685+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f9444051ba0 con 0x7f944c109bf0 2024-03-24T11:39:33.901 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug Started the configuration object watcher 2024-03-24T11:39:33.901 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug Checking for config object changes every 1s 2024-03-24T11:39:33.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug Processing osd blocklist entries for this node 2024-03-24T11:39:33.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.907+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f9438000bc0 con 0x7f943406e770 2024-03-24T11:39:33.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.914+0000 7f9433fff700 1 -- 172.21.15.119:0/2487228771 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (unknown 0 0 1342662408) 0x7f9438000bc0 con 0x7f943406e770 2024-03-24T11:39:33.915 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled prometheus update... 2024-03-24T11:39:33.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.917+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 >> v1:172.21.15.114:6800/3103139641 conn(0x7f943406e770 legacy=0x7f9434070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:33.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.917+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 >> v1:172.21.15.114:6789/0 conn(0x7f944c109bf0 legacy=0x7f944c1a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:33.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.917+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 shutdown_connections 2024-03-24T11:39:33.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.917+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 >> 172.21.15.119:0/2487228771 conn(0x7f944c0fbf70 msgr2=0x7f944c10c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:33.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.917+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 shutdown_connections 2024-03-24T11:39:33.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:33.917+0000 7f9451311700 1 -- 172.21.15.119:0/2487228771 wait complete. 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: pgmap v201: 132 pgs: 7 creating+activating, 125 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/1032161254' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:39:34.169 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:34 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.169 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug Reading the configuration object to update local LIO configuration 2024-03-24T11:39:34.170 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug Configuration does not have an entry for this host(smithi119.front.sepia.ceph.com) - nothing to define to LIO 2024-03-24T11:39:34.170 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: pgmap v201: 132 pgs: 7 creating+activating, 125 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/1032161254' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:39:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: pgmap v201: 132 pgs: 7 creating+activating, 125 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/1032161254' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:39:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:34 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:34.794 DEBUG:teuthology.orchestra.run.smithi119:prometheus.a> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T11:39:34.796 INFO:tasks.cephadm:Adding node-exporter.a on smithi114 2024-03-24T11:39:34.796 INFO:tasks.cephadm:Adding node-exporter.b on smithi119 2024-03-24T11:39:34.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi114=a;smithi119=b' 2024-03-24T11:39:35.375 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:35 smithi119 ceph-mon[33714]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:39:35.375 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:35 smithi119 ceph-mon[33714]: from='client.14565 v1:172.21.15.119:0/2487228771' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:35.375 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:35 smithi119 ceph-mon[33714]: Saving service prometheus spec with placement smithi119=a;count:1 2024-03-24T11:39:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:35 smithi114 ceph-mon[30312]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:39:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:35 smithi114 ceph-mon[30312]: from='client.14565 v1:172.21.15.119:0/2487228771' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:35 smithi114 ceph-mon[30312]: Saving service prometheus spec with placement smithi119=a;count:1 2024-03-24T11:39:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:35 smithi114 ceph-mon[42466]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:39:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:35 smithi114 ceph-mon[42466]: from='client.14565 v1:172.21.15.119:0/2487228771' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:35 smithi114 ceph-mon[42466]: Saving service prometheus spec with placement smithi119=a;count:1 2024-03-24T11:39:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:36.399 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:36 smithi119 ceph-mon[33714]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.2 KiB/s wr, 167 op/s 2024-03-24T11:39:36.400 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:36 smithi119 ceph-mon[33714]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:39:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:36 smithi114 ceph-mon[30312]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.2 KiB/s wr, 167 op/s 2024-03-24T11:39:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:36 smithi114 ceph-mon[30312]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:39:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:36 smithi114 ceph-mon[42466]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.2 KiB/s wr, 167 op/s 2024-03-24T11:39:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:36 smithi114 ceph-mon[42466]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:39:37.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.213+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1594434659 >> v1:172.21.15.119:6789/0 conn(0x7f7b34108ad0 legacy=0x7f7b3410af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:37.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.213+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1594434659 shutdown_connections 2024-03-24T11:39:37.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.213+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1594434659 >> 172.21.15.119:0/1594434659 conn(0x7f7b340fbf00 msgr2=0x7f7b340fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:37.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.213+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1594434659 shutdown_connections 2024-03-24T11:39:37.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.213+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1594434659 wait complete. 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b3aa92700 1 Processor -- start 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b3aa92700 1 -- start start 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b3aa92700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7b341979d0 con 0x7f7b34108ad0 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b3aa92700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7b341a31c0 con 0x7f7b34104d90 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b3aa92700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7b341a4360 con 0x7f7b34100960 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b33fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7b34104d90 0x7f7b3419d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48872/0 (socket says 172.21.15.119:48872) 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b33fff700 1 -- 172.21.15.119:0/1452922401 learned_addr learned my addr 172.21.15.119:0/1452922401 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.214+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302381645 0 0) 0x7f7b341a31c0 con 0x7f7b34104d90 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b240035b0 con 0x7f7b34104d90 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759357722 0 0) 0x7f7b341979d0 con 0x7f7b34108ad0 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b341a31c0 con 0x7f7b34108ad0 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743750750 0 0) 0x7f7b341a4360 con 0x7f7b34100960 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b341979d0 con 0x7f7b34100960 2024-03-24T11:39:37.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3940860507 0 0) 0x7f7b240035b0 con 0x7f7b34104d90 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b341a4360 con 0x7f7b34104d90 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f7b200018a0 con 0x7f7b34104d90 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3739369346 0 0) 0x7f7b341a31c0 con 0x7f7b34108ad0 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b240035b0 con 0x7f7b34108ad0 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f7b2c001e20 con 0x7f7b34108ad0 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008391878 0 0) 0x7f7b341979d0 con 0x7f7b34100960 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b341a31c0 con 0x7f7b34100960 2024-03-24T11:39:37.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f7b280017a0 con 0x7f7b34100960 2024-03-24T11:39:37.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1227288695 0 0) 0x7f7b341a4360 con 0x7f7b34104d90 2024-03-24T11:39:37.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 >> v1:172.21.15.114:6790/0 conn(0x7f7b34100960 legacy=0x7f7b34196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:37.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 >> v1:172.21.15.114:6789/0 conn(0x7f7b34108ad0 legacy=0x7f7b341a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:37.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b341a5500 con 0x7f7b34104d90 2024-03-24T11:39:37.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b341a2210 con 0x7f7b34104d90 2024-03-24T11:39:37.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f7b20003b30 con 0x7f7b34104d90 2024-03-24T11:39:37.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f7b20005c30 con 0x7f7b34104d90 2024-03-24T11:39:37.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.215+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b341a2710 con 0x7f7b34104d90 2024-03-24T11:39:37.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.216+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b34068560 con 0x7f7b34104d90 2024-03-24T11:39:37.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.217+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 16) v1 ==== 91097+0+0 (unknown 2740255868 0 0) 0x7f7b200036a0 con 0x7f7b34104d90 2024-03-24T11:39:37.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.217+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 1359784666 0 0) 0x7f7b20088300 con 0x7f7b34104d90 2024-03-24T11:39:37.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.222+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f7b20051cc0 con 0x7f7b34104d90 2024-03-24T11:39:37.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.430+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi119=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b34107910 con 0x7f7b2406e690 2024-03-24T11:39:37.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.440+0000 7f7b31ffb700 1 -- 172.21.15.119:0/1452922401 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (unknown 0 0 240551134) 0x7f7b34107910 con 0x7f7b2406e690 2024-03-24T11:39:37.441 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled node-exporter update... 2024-03-24T11:39:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.442+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 >> v1:172.21.15.114:6800/3103139641 conn(0x7f7b2406e690 legacy=0x7f7b24070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.442+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 >> v1:172.21.15.119:6789/0 conn(0x7f7b34104d90 legacy=0x7f7b3419d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.442+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 shutdown_connections 2024-03-24T11:39:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.442+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 >> 172.21.15.119:0/1452922401 conn(0x7f7b340fbf00 msgr2=0x7f7b3410bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.442+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 shutdown_connections 2024-03-24T11:39:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:37.442+0000 7f7b3aa92700 1 -- 172.21.15.119:0/1452922401 wait complete. 2024-03-24T11:39:38.353 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:38 smithi119 ceph-mon[33714]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-24T11:39:38.353 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:38 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:38.353 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:38 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:38 smithi114 ceph-mon[30312]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-24T11:39:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:38 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:38 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:38 smithi114 ceph-mon[42466]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-24T11:39:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:38 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:38 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:38.672 DEBUG:teuthology.orchestra.run.smithi114:node-exporter.a> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T11:39:38.675 DEBUG:teuthology.orchestra.run.smithi119:node-exporter.b> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T11:39:38.677 INFO:tasks.cephadm:Adding alertmanager.a on smithi114 2024-03-24T11:39:38.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi114=a' 2024-03-24T11:39:38.964 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:39.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:39 smithi119 ceph-mon[33714]: from='client.24466 v1:172.21.15.119:0/1452922401' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi119=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:39.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:39 smithi119 ceph-mon[33714]: Saving service node-exporter spec with placement smithi114=a;smithi119=b;count:2 2024-03-24T11:39:39.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:39.356 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:39.357 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:39 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[30312]: from='client.24466 v1:172.21.15.119:0/1452922401' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi119=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:39.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[30312]: Saving service node-exporter spec with placement smithi114=a;smithi119=b;count:2 2024-03-24T11:39:39.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:39.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:39.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[42466]: from='client.24466 v1:172.21.15.119:0/1452922401' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi119=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:39.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[42466]: Saving service node-exporter spec with placement smithi114=a;smithi119=b;count:2 2024-03-24T11:39:39.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:39.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:39.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:39.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:39 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:40.254 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:40 smithi119 ceph-mon[33714]: Deploying daemon prometheus.a on smithi119 2024-03-24T11:39:40.255 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:40 smithi119 ceph-mon[33714]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.6 KiB/s wr, 116 op/s 2024-03-24T11:39:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:40 smithi114 ceph-mon[30312]: Deploying daemon prometheus.a on smithi119 2024-03-24T11:39:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:40 smithi114 ceph-mon[30312]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.6 KiB/s wr, 116 op/s 2024-03-24T11:39:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:40 smithi114 ceph-mon[42466]: Deploying daemon prometheus.a on smithi119 2024-03-24T11:39:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:40 smithi114 ceph-mon[42466]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.6 KiB/s wr, 116 op/s 2024-03-24T11:39:41.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.980+0000 7fe4e4086700 1 -- 172.21.15.119:0/533206891 >> v1:172.21.15.119:6789/0 conn(0x7fe4dc101a90 legacy=0x7fe4dc101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:41.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.980+0000 7fe4e4086700 1 -- 172.21.15.119:0/533206891 shutdown_connections 2024-03-24T11:39:41.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.980+0000 7fe4e4086700 1 -- 172.21.15.119:0/533206891 >> 172.21.15.119:0/533206891 conn(0x7fe4dc0fd070 msgr2=0x7fe4dc0ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:41.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.980+0000 7fe4e4086700 1 -- 172.21.15.119:0/533206891 shutdown_connections 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.980+0000 7fe4e4086700 1 -- 172.21.15.119:0/533206891 wait complete. 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4e4086700 1 Processor -- start 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4e4086700 1 -- start start 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4e4086700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4dc1a40d0 con 0x7fe4dc109c00 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4e4086700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4dc1a52d0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4e4086700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4dc1a64d0 con 0x7fe4dc101a90 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4e1621700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe4dc105ec0 0x7fe4dc19eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46522/0 (socket says 172.21.15.119:46522) 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4e1621700 1 -- 172.21.15.119:0/2489913485 learned_addr learned my addr 172.21.15.119:0/2489913485 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736300208 0 0) 0x7fe4dc1a52d0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c80035b0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219040301 0 0) 0x7fe4dc1a40d0 con 0x7fe4dc109c00 2024-03-24T11:39:41.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4dc1a52d0 con 0x7fe4dc109c00 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221869713 0 0) 0x7fe4dc1a64d0 con 0x7fe4dc101a90 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.981+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4dc1a40d0 con 0x7fe4dc101a90 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4280612318 0 0) 0x7fe4c80035b0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4dc1a64d0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe4d80018a0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371311195 0 0) 0x7fe4dc1a52d0 con 0x7fe4dc109c00 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4c80035b0 con 0x7fe4dc109c00 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe4d4001b70 con 0x7fe4dc109c00 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3836820210 0 0) 0x7fe4dc1a40d0 con 0x7fe4dc101a90 2024-03-24T11:39:41.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4dc1a52d0 con 0x7fe4dc101a90 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe4cc0018c0 con 0x7fe4dc101a90 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3992796562 0 0) 0x7fe4dc1a64d0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 >> v1:172.21.15.114:6790/0 conn(0x7fe4dc101a90 legacy=0x7fe4dc075930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 >> v1:172.21.15.114:6789/0 conn(0x7fe4dc109c00 legacy=0x7fe4dc1a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc1a76d0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc1a42c0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc1a47c0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fe4d8003b30 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.982+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe4d8005bb0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.983+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c0005150 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.983+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 16) v1 ==== 91097+0+0 (unknown 2740255868 0 0) 0x7fe4d80036a0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.984+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 1359784666 0 0) 0x7fe4d8088280 con 0x7fe4dc105ec0 2024-03-24T11:39:41.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:40.988+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fe4d8051bc0 con 0x7fe4dc105ec0 2024-03-24T11:39:41.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.196+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4c0000bc0 con 0x7fe4c806e5c0 2024-03-24T11:39:41.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.203+0000 7fe4d2ffd700 1 -- 172.21.15.119:0/2489913485 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (unknown 0 0 1850065467) 0x7fe4c0000bc0 con 0x7fe4c806e5c0 2024-03-24T11:39:41.204 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled alertmanager update... 2024-03-24T11:39:41.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.205+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 >> v1:172.21.15.114:6800/3103139641 conn(0x7fe4c806e5c0 legacy=0x7fe4c8070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:41.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.205+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 >> v1:172.21.15.119:6789/0 conn(0x7fe4dc105ec0 legacy=0x7fe4dc19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:41.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.205+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 shutdown_connections 2024-03-24T11:39:41.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.205+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 >> 172.21.15.119:0/2489913485 conn(0x7fe4dc0fd070 msgr2=0x7fe4dc108310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:41.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.205+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 shutdown_connections 2024-03-24T11:39:41.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:41.205+0000 7fe4e4086700 1 -- 172.21.15.119:0/2489913485 wait complete. 2024-03-24T11:39:42.176 DEBUG:teuthology.orchestra.run.smithi114:alertmanager.a> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T11:39:42.179 INFO:tasks.cephadm:Adding grafana.a on smithi119 2024-03-24T11:39:42.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi119=a' 2024-03-24T11:39:42.423 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:42 smithi114 ceph-mon[30312]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-24T11:39:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:42 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:42 smithi114 ceph-mon[42466]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-24T11:39:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:42 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:42.544 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:42 smithi119 ceph-mon[33714]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-24T11:39:42.544 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:42 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:43 smithi119 ceph-mon[33714]: from='client.24472 v1:172.21.15.119:0/2489913485' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:43 smithi119 ceph-mon[33714]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-03-24T11:39:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:43 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:43 smithi114 ceph-mon[42466]: from='client.24472 v1:172.21.15.119:0/2489913485' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:43 smithi114 ceph-mon[42466]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-03-24T11:39:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:43 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:43 smithi114 ceph-mon[30312]: from='client.24472 v1:172.21.15.119:0/2489913485' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:43 smithi114 ceph-mon[30312]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-03-24T11:39:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:43 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:44.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:39:43 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:39:44.432 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:44 smithi119 ceph-mon[33714]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 818 B/s wr, 23 op/s 2024-03-24T11:39:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:44 smithi114 ceph-mon[30312]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 818 B/s wr, 23 op/s 2024-03-24T11:39:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:44 smithi114 ceph-mon[42466]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 818 B/s wr, 23 op/s 2024-03-24T11:39:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:45 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:45 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:45 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:46 smithi119 ceph-mon[33714]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 180 B/s wr, 2 op/s 2024-03-24T11:39:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:46 smithi114 ceph-mon[30312]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 180 B/s wr, 2 op/s 2024-03-24T11:39:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:46 smithi114 ceph-mon[42466]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 180 B/s wr, 2 op/s 2024-03-24T11:39:47.381 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.381 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.381 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:47 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.379+0000 7f340092d700 1 -- 172.21.15.119:0/3092428411 >> v1:172.21.15.119:6789/0 conn(0x7f33fc101a80 legacy=0x7f33fc101ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.380+0000 7f340092d700 1 -- 172.21.15.119:0/3092428411 shutdown_connections 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.380+0000 7f340092d700 1 -- 172.21.15.119:0/3092428411 >> 172.21.15.119:0/3092428411 conn(0x7f33fc0fd020 msgr2=0x7f33fc0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.380+0000 7f340092d700 1 -- 172.21.15.119:0/3092428411 shutdown_connections 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.380+0000 7f340092d700 1 -- 172.21.15.119:0/3092428411 wait complete. 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.380+0000 7f340092d700 1 Processor -- start 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.380+0000 7f340092d700 1 -- start start 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f340092d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33fc1a3db0 con 0x7f33fc109bf0 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f340092d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33fc1a4fb0 con 0x7f33fc105eb0 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f340092d700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33fc1a61b0 con 0x7f33fc101a80 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f1d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f33fc105eb0 0x7f33fc19eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46538/0 (socket says 172.21.15.119:46538) 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f1d9b700 1 -- 172.21.15.119:0/3370459230 learned_addr learned my addr 172.21.15.119:0/3370459230 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:47.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976549767 0 0) 0x7f33fc1a4fb0 con 0x7f33fc105eb0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e80035b0 con 0x7f33fc105eb0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079954719 0 0) 0x7f33fc1a3db0 con 0x7f33fc109bf0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc1a4fb0 con 0x7f33fc109bf0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049441928 0 0) 0x7f33fc1a61b0 con 0x7f33fc101a80 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc1a3db0 con 0x7f33fc101a80 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164000954 0 0) 0x7f33e80035b0 con 0x7f33fc105eb0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33fc1a61b0 con 0x7f33fc105eb0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f33e40019b0 con 0x7f33fc105eb0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2672334206 0 0) 0x7f33fc1a4fb0 con 0x7f33fc109bf0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33e80035b0 con 0x7f33fc109bf0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f33f4001ae0 con 0x7f33fc109bf0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3922847055 0 0) 0x7f33fc1a3db0 con 0x7f33fc101a80 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33fc1a4fb0 con 0x7f33fc101a80 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f33ec004440 con 0x7f33fc101a80 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3718453095 0 0) 0x7f33fc1a61b0 con 0x7f33fc105eb0 2024-03-24T11:39:47.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 >> v1:172.21.15.114:6790/0 conn(0x7f33fc101a80 legacy=0x7f33fc075930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 >> v1:172.21.15.114:6789/0 conn(0x7f33fc109bf0 legacy=0x7f33fc1a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc1a73b0 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.381+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc1a3fa0 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.382+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc1a44a0 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.382+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f33e4003de0 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.382+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f33e4005e40 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.382+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc005150 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.383+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 16) v1 ==== 91097+0+0 (unknown 2740255868 0 0) 0x7f33e401c410 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.383+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6001+0+0 (unknown 4173716926 0 0) 0x7f33e4088550 con 0x7f33fc105eb0 2024-03-24T11:39:47.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.388+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f33e4051c40 con 0x7f33fc105eb0 2024-03-24T11:39:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:47 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:47 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:39:47.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.596+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 --> v1:172.21.15.114:6800/3103139641 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f33dc000bc0 con 0x7f33e806e520 2024-03-24T11:39:47.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.603+0000 7f33f3fff700 1 -- 172.21.15.119:0/3370459230 <== mgr.14152 v1:172.21.15.114:6800/3103139641 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (unknown 0 0 664801700) 0x7f33dc000bc0 con 0x7f33e806e520 2024-03-24T11:39:47.604 INFO:teuthology.orchestra.run.smithi119.stdout:Scheduled grafana update... 2024-03-24T11:39:47.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.606+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 >> v1:172.21.15.114:6800/3103139641 conn(0x7f33e806e520 legacy=0x7f33e80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.606+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 >> v1:172.21.15.119:6789/0 conn(0x7f33fc105eb0 legacy=0x7f33fc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.606+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 shutdown_connections 2024-03-24T11:39:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.606+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 >> 172.21.15.119:0/3370459230 conn(0x7f33fc0fd020 msgr2=0x7f33fc10ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.606+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 shutdown_connections 2024-03-24T11:39:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:47.606+0000 7f340092d700 1 -- 172.21.15.119:0/3370459230 wait complete. 2024-03-24T11:39:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[30312]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-24T11:39:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T11:39:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:39:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[30312]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:39:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[42466]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-24T11:39:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T11:39:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:39:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[42466]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:39:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:48 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:48 smithi119 ceph-mon[33714]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-24T11:39:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T11:39:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:39:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:48 smithi119 ceph-mon[33714]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:39:48.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:48 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:49.399 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:49 smithi119 ceph-mon[33714]: from='client.24478 v1:172.21.15.119:0/3370459230' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:49.399 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:49 smithi119 ceph-mon[33714]: Saving service grafana spec with placement smithi119=a;count:1 2024-03-24T11:39:49.399 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:49 smithi119 ceph-mon[33714]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:39:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:49 smithi114 ceph-mon[30312]: from='client.24478 v1:172.21.15.119:0/3370459230' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:49 smithi114 ceph-mon[30312]: Saving service grafana spec with placement smithi119=a;count:1 2024-03-24T11:39:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:49 smithi114 ceph-mon[30312]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:39:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:49 smithi114 ceph-mon[42466]: from='client.24478 v1:172.21.15.119:0/3370459230' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:49 smithi114 ceph-mon[42466]: Saving service grafana spec with placement smithi119=a;count:1 2024-03-24T11:39:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:49 smithi114 ceph-mon[42466]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:39:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:50 smithi119 ceph-mon[33714]: pgmap v212: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:39:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:50 smithi114 ceph-mon[30312]: pgmap v212: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:39:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:50 smithi114 ceph-mon[42466]: pgmap v212: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:39:50.742 DEBUG:teuthology.orchestra.run.smithi119:grafana.a> sudo journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T11:39:50.745 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:39:50.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:39:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:51.804 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:51 smithi119 systemd[1]: Starting Ceph prometheus.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:39:52.232 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-mon[33714]: pgmap v213: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:39:52.233 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:51 smithi119 podman[64178]: 2024-03-24 11:39:51.858413209 +0000 UTC m=+0.018960574 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T11:39:52.233 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:51 smithi119 podman[64178]: 2024-03-24 11:39:51.964203439 +0000 UTC m=+0.124750810 container create e8f9fae1104f9ff133d0a9e9f3b59ebc5071ee15fea319c05a39fb05f7ce2eb9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:52.233 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 podman[64178]: 2024-03-24 11:39:52.232859501 +0000 UTC m=+0.393406866 container init e8f9fae1104f9ff133d0a9e9f3b59ebc5071ee15fea319c05a39fb05f7ce2eb9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:52 smithi114 ceph-mon[30312]: pgmap v213: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:39:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:52 smithi114 ceph-mon[42466]: pgmap v213: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 podman[64178]: 2024-03-24 11:39:52.238138138 +0000 UTC m=+0.398685518 container start e8f9fae1104f9ff133d0a9e9f3b59ebc5071ee15fea319c05a39fb05f7ce2eb9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.261Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.261Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.261Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi119 (none))" 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.261Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.261Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.263Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.263Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.265Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.265Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.266Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:39:52.555 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.266Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.514µs 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.266Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.266Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.266Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=16.39µs wal_replay_duration=191.935µs wbl_replay_duration=107ns total_replay_duration=228.251µs 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.267Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.267Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.267Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.284Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.705832ms db_storage=645ns remote_storage=1.024µs web_handler=213ns query_engine=914ns scrape=562.335µs scrape_sd=63.411µs notify=728ns notify_sd=4.276µs rules=15.90862ms tracing=7.19µs 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.284Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:39:52.284Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 bash[64178]: e8f9fae1104f9ff133d0a9e9f3b59ebc5071ee15fea319c05a39fb05f7ce2eb9 2024-03-24T11:39:52.556 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:39:52 smithi119 systemd[1]: Started Ceph prometheus.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:39:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.867+0000 7f2198482700 1 -- 172.21.15.114:0/163273398 >> v1:172.21.15.114:6790/0 conn(0x7f2190101f60 legacy=0x7f2190102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.867+0000 7f2198482700 1 -- 172.21.15.114:0/163273398 shutdown_connections 2024-03-24T11:39:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.867+0000 7f2198482700 1 -- 172.21.15.114:0/163273398 >> 172.21.15.114:0/163273398 conn(0x7f21900fb6a0 msgr2=0x7f21900fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.867+0000 7f2198482700 1 -- 172.21.15.114:0/163273398 shutdown_connections 2024-03-24T11:39:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.867+0000 7f2198482700 1 -- 172.21.15.114:0/163273398 wait complete. 2024-03-24T11:39:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.868+0000 7f2198482700 1 Processor -- start 2024-03-24T11:39:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.868+0000 7f2198482700 1 -- start start 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.868+0000 7f2198482700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2190196700 con 0x7f2190101f60 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.868+0000 7f2198482700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21901a1ef0 con 0x7f2190104e50 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.868+0000 7f2198482700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21901a3090 con 0x7f2190108ab0 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.868+0000 7f219621e700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2190101f60 0x7f2190195bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41440/0 (socket says 172.21.15.114:41440) 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.868+0000 7f219621e700 1 -- 172.21.15.114:0/2078841406 learned_addr learned my addr 172.21.15.114:0/2078841406 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285632656 0 0) 0x7f21901a3090 con 0x7f2190108ab0 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f217c0035b0 con 0x7f2190108ab0 2024-03-24T11:39:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891461870 0 0) 0x7f2190196700 con 0x7f2190101f60 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21901a3090 con 0x7f2190101f60 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386381697 0 0) 0x7f21901a1ef0 con 0x7f2190104e50 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2190196700 con 0x7f2190104e50 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1847265210 0 0) 0x7f217c0035b0 con 0x7f2190108ab0 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21901a1ef0 con 0x7f2190108ab0 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843100726 0 0) 0x7f21901a3090 con 0x7f2190101f60 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f217c0035b0 con 0x7f2190101f60 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2188001b70 con 0x7f2190108ab0 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f218c004430 con 0x7f2190101f60 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1581518142 0 0) 0x7f2190196700 con 0x7f2190104e50 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21901a3090 con 0x7f2190104e50 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f21800017d0 con 0x7f2190104e50 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2538269714 0 0) 0x7f21901a1ef0 con 0x7f2190108ab0 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 >> v1:172.21.15.119:6789/0 conn(0x7f2190104e50 legacy=0x7f219019c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 >> v1:172.21.15.114:6789/0 conn(0x7f2190101f60 legacy=0x7f2190195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21901a4230 con 0x7f2190108ab0 2024-03-24T11:39:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21901a0f40 con 0x7f2190108ab0 2024-03-24T11:39:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21901a1440 con 0x7f2190108ab0 2024-03-24T11:39:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2188001d60 con 0x7f2190108ab0 2024-03-24T11:39:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.869+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2188004d50 con 0x7f2190108ab0 2024-03-24T11:39:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.870+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2174005150 con 0x7f2190108ab0 2024-03-24T11:39:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.871+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 16) v1 ==== 91097+0+0 (unknown 2740255868 0 0) 0x7f2188003a00 con 0x7f2190108ab0 2024-03-24T11:39:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.871+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (unknown 2262200551 0 0) 0x7f2188087500 con 0x7f2190108ab0 2024-03-24T11:39:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:52.876+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f2188050bf0 con 0x7f2190108ab0 2024-03-24T11:39:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.151+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f2174005400 con 0x7f2190108ab0 2024-03-24T11:39:53.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.155+0000 7f21877fe700 1 -- 172.21.15.114:0/2078841406 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (unknown 57899324 0 745542118) 0x7f2188052cf0 con 0x7f2190108ab0 2024-03-24T11:39:53.156 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-24T11:39:53.156 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAJEQBmC6AaCRAA3r227th095NOQ3+NfKB3pg== 2024-03-24T11:39:53.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.157+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 >> v1:172.21.15.114:6800/3103139641 conn(0x7f217c0707a0 legacy=0x7f217c072c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:53.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.157+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 >> v1:172.21.15.114:6790/0 conn(0x7f2190108ab0 legacy=0x7f219019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:53.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.157+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 shutdown_connections 2024-03-24T11:39:53.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.157+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 >> 172.21.15.114:0/2078841406 conn(0x7f21900fb6a0 msgr2=0x7f21900fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:53.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.157+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 shutdown_connections 2024-03-24T11:39:53.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:53.157+0000 7f2198482700 1 -- 172.21.15.114:0/2078841406 wait complete. 2024-03-24T11:39:53.424 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.424 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.424 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.424 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:39:53.424 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68 B/s, 1 objects/s recovering 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2078841406' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: from='client.34466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[42466]: from='client.34466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68 B/s, 1 objects/s recovering 2024-03-24T11:39:53.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2078841406' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:53.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: from='client.34466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:53.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-mon[30312]: from='client.34466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:39:53.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ignoring --setuser ceph since I am not root 2024-03-24T11:39:53.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ignoring --setgroup ceph since I am not root 2024-03-24T11:39:53.804 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: ignoring --setuser ceph since I am not root 2024-03-24T11:39:53.804 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: ignoring --setgroup ceph since I am not root 2024-03-24T11:39:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' 2024-03-24T11:39:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:39:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68 B/s, 1 objects/s recovering 2024-03-24T11:39:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2078841406' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: from='client.34466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-mon[33714]: from='client.34466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:39:53.938 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:39:53.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:39:53.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:39:53.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:39:54.238 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:53 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:53.959+0000 7f38682ea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:39:54.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:54.056+0000 7f38682ea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:39:54.250 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:39:54.304 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:53.962+0000 7f9ea6feb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:39:54.304 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:54 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:54.060+0000 7f9ea6feb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:39:54.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:54 smithi119 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:39:54.650 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:54 smithi119 ceph-mon[33714]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:39:54.699 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-mon[30312]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:39:54.699 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-mon[30312]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:39:54.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-mon[42466]: from='mgr.14152 v1:172.21.15.114:0/2761043132' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:39:54.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-mon[42466]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:39:54.912 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:54 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:54.750+0000 7f9ea6feb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:39:54.912 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:54 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:54.847+0000 7f9ea6feb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:39:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:54.698+0000 7f38682ea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:39:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:54.792+0000 7f38682ea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:39:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:54.933+0000 7f38682ea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:39:55.168 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:54 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:54.989+0000 7f9ea6feb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:39:55.168 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:55.085+0000 7f9ea6feb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:39:55.168 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:55.167+0000 7f9ea6feb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:39:55.377 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:55.029+0000 7f38682ea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:39:55.377 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:55.113+0000 7f38682ea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:39:55.377 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:55.376+0000 7f38682ea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:39:55.667 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:55.486+0000 7f38682ea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:55.667 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:55.571+0000 7f38682ea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:39:55.667 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:55.666+0000 7f38682ea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:39:55.729 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:55.434+0000 7f9ea6feb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:39:55.730 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:55.544+0000 7f9ea6feb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:55.730 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:55.635+0000 7f9ea6feb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:39:55.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:55.843+0000 7f38682ea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:39:56.010 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:55.732+0000 7f9ea6feb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:39:56.010 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:55.915+0000 7f9ea6feb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:39:56.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.123+0000 7f829f06c700 1 -- 172.21.15.119:0/1116472631 >> v1:172.21.15.119:6789/0 conn(0x7f8298100960 legacy=0x7f8298100d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:56.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.123+0000 7f829f06c700 1 -- 172.21.15.119:0/1116472631 shutdown_connections 2024-03-24T11:39:56.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.123+0000 7f829f06c700 1 -- 172.21.15.119:0/1116472631 >> 172.21.15.119:0/1116472631 conn(0x7f82980fbf40 msgr2=0x7f82980fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:56.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.123+0000 7f829f06c700 1 -- 172.21.15.119:0/1116472631 shutdown_connections 2024-03-24T11:39:56.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.123+0000 7f829f06c700 1 -- 172.21.15.119:0/1116472631 wait complete. 2024-03-24T11:39:56.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.124+0000 7f829f06c700 1 Processor -- start 2024-03-24T11:39:56.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.124+0000 7f829f06c700 1 -- start start 2024-03-24T11:39:56.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.124+0000 7f829f06c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f829810d800 con 0x7f8298104d90 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.124+0000 7f829f06c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8298196ab0 con 0x7f8298100960 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.124+0000 7f829f06c700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82981a3250 con 0x7f8298108ad0 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.124+0000 7f829ce08700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8298100960 0x7f8298195f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59466/0 (socket says 172.21.15.119:59466) 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.124+0000 7f829ce08700 1 -- 172.21.15.119:0/3296515197 learned_addr learned my addr 172.21.15.119:0/3296515197 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334107344 0 0) 0x7f8298196ab0 con 0x7f8298100960 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82880035b0 con 0x7f8298100960 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165073129 0 0) 0x7f82981a3250 con 0x7f8298108ad0 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8298196ab0 con 0x7f8298108ad0 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716543032 0 0) 0x7f829810d800 con 0x7f8298104d90 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82981a3250 con 0x7f8298104d90 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3453771354 0 0) 0x7f8298196ab0 con 0x7f8298108ad0 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f829810d800 con 0x7f8298108ad0 2024-03-24T11:39:56.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f8290001c30 con 0x7f8298108ad0 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387030720 0 0) 0x7f82880035b0 con 0x7f8298100960 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8298196ab0 con 0x7f8298100960 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f828c001880 con 0x7f8298100960 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1497009203 0 0) 0x7f82981a3250 con 0x7f8298104d90 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82880035b0 con 0x7f8298104d90 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f82840017f0 con 0x7f8298104d90 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3640002731 0 0) 0x7f829810d800 con 0x7f8298108ad0 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 >> v1:172.21.15.119:6789/0 conn(0x7f8298100960 legacy=0x7f8298195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 >> v1:172.21.15.114:6789/0 conn(0x7f8298104d90 legacy=0x7f829819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.125+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82981a43f0 con 0x7f8298108ad0 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.126+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82981a33e0 con 0x7f8298108ad0 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.126+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f8290003e90 con 0x7f8298108ad0 2024-03-24T11:39:56.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.126+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f8290004d70 con 0x7f8298108ad0 2024-03-24T11:39:56.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.126+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82981a38e0 con 0x7f8298108ad0 2024-03-24T11:39:56.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.127+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8298068560 con 0x7f8298108ad0 2024-03-24T11:39:56.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.127+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 17) v1 ==== 91111+0+0 (unknown 3770792243 0 0) 0x7f82900060e0 con 0x7f8298108ad0 2024-03-24T11:39:56.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.128+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (unknown 2262200551 0 0) 0x7f82900875c0 con 0x7f8298108ad0 2024-03-24T11:39:56.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.128+0000 7f829ce08700 1 -- 172.21.15.119:0/3296515197 >> v1:172.21.15.114:6800/3103139641 conn(0x7f8288070850 legacy=0x7f8288072d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/3103139641 2024-03-24T11:39:56.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.132+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f8290050e80 con 0x7f8298108ad0 2024-03-24T11:39:56.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.329+0000 7f829ce08700 1 -- 172.21.15.119:0/3296515197 >> v1:172.21.15.114:6800/3103139641 conn(0x7f8288070850 legacy=0x7f8288072d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/3103139641 2024-03-24T11:39:56.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.449+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f82981a3b90 con 0x7f8298108ad0 2024-03-24T11:39:56.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.455+0000 7f8295ffb700 1 -- 172.21.15.119:0/3296515197 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (unknown 3508017507 0 1141691648) 0x7f8290052db0 con 0x7f8298108ad0 2024-03-24T11:39:56.456 INFO:teuthology.orchestra.run.smithi119.stdout:[client.1] 2024-03-24T11:39:56.456 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAMEQBmXZnfGhAAwbPoDAjhQLhfEMrr0Jw66A== 2024-03-24T11:39:56.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.458+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 >> v1:172.21.15.114:6800/3103139641 conn(0x7f8288070850 legacy=0x7f8288072d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:39:56.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.458+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 >> v1:172.21.15.114:6790/0 conn(0x7f8298108ad0 legacy=0x7f829819fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:56.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.458+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 shutdown_connections 2024-03-24T11:39:56.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.458+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 >> 172.21.15.119:0/3296515197 conn(0x7f82980fbf40 msgr2=0x7f82980fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:56.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.458+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 shutdown_connections 2024-03-24T11:39:56.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-24T11:39:56.458+0000 7f829f06c700 1 -- 172.21.15.119:0/3296515197 wait complete. 2024-03-24T11:39:56.772 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.119:0/3296515197' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:56.772 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:56 smithi119 ceph-mon[33714]: from='client.34481 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:56.772 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:56 smithi119 ceph-mon[33714]: from='client.34481 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:39:56.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.119:0/3296515197' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:56.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-mon[30312]: from='client.34481 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:56.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-mon[30312]: from='client.34481 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:39:56.805 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.119:0/3296515197' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:56.805 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-mon[42466]: from='client.34481 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:39:56.805 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-mon[42466]: from='client.34481 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:39:56.805 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:56.804+0000 7f38682ea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:39:57.052 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:56 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:56.919+0000 7f9ea6feb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:39:57.100 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-24T11:39:57.100 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T11:39:57.100 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T11:39:57.144 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:39:57.144 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:39:57.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:39:57.169 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:56 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:56.935+0000 7f38682ea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:39:57.304 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:57 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:57.051+0000 7f9ea6feb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:39:57.392 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:57.488 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:57 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:57.275+0000 7f38682ea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:39:57.489 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:57 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:57.376+0000 7f38682ea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:39:57.680 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:57 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:57.393+0000 7f9ea6feb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:39:57.680 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:57 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:57.483+0000 7f9ea6feb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:39:57.680 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:57 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:57.679+0000 7f9ea6feb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:57.780 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:57 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:57.585+0000 7f38682ea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:57.780 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:57 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:57.779+0000 7f38682ea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:39:58.054 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:57 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:57.864+0000 7f9ea6feb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:39:58.331 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:58 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:58.155+0000 7f9ea6feb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:58.339 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:58 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:58.075+0000 7f38682ea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:58.592 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:58 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:58.339+0000 7f38682ea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:58.676 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:58 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:58.428+0000 7f9ea6feb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:58.846 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:58 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:58.591+0000 7f38682ea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:39:59.054 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:58 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:58.675+0000 7f9ea6feb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:39:59.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.075+0000 7fb114fae700 1 -- 172.21.15.114:0/4091217733 >> v1:172.21.15.114:6789/0 conn(0x7fb110106900 legacy=0x7fb110106ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:59.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.076+0000 7fb114fae700 1 -- 172.21.15.114:0/4091217733 shutdown_connections 2024-03-24T11:39:59.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.076+0000 7fb114fae700 1 -- 172.21.15.114:0/4091217733 >> 172.21.15.114:0/4091217733 conn(0x7fb1100fc410 msgr2=0x7fb1100fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:59.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.076+0000 7fb114fae700 1 -- 172.21.15.114:0/4091217733 shutdown_connections 2024-03-24T11:39:59.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.076+0000 7fb114fae700 1 -- 172.21.15.114:0/4091217733 wait complete. 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb114fae700 1 Processor -- start 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb114fae700 1 -- start start 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb114fae700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1101a6910 con 0x7fb110106900 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb114fae700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1101a7af0 con 0x7fb1101097b0 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb114fae700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1101a8cf0 con 0x7fb11010d380 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb10e59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb110106900 0x7fb11010cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41478/0 (socket says 172.21.15.114:41478) 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb10e59c700 1 -- 172.21.15.114:0/3951120 learned_addr learned my addr 172.21.15.114:0/3951120 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359999050 0 0) 0x7fb1101a6910 con 0x7fb110106900 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f80035b0 con 0x7fb110106900 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176822309 0 0) 0x7fb1101a8cf0 con 0x7fb11010d380 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1101a6910 con 0x7fb11010d380 2024-03-24T11:39:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129840016 0 0) 0x7fb1101a7af0 con 0x7fb1101097b0 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.077+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1101a8cf0 con 0x7fb1101097b0 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368663479 0 0) 0x7fb1101a6910 con 0x7fb11010d380 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1101a7af0 con 0x7fb11010d380 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 759639279 0 0) 0x7fb0f80035b0 con 0x7fb110106900 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1101a6910 con 0x7fb110106900 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb108001bf0 con 0x7fb11010d380 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb100004460 con 0x7fb110106900 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3883463774 0 0) 0x7fb1101a8cf0 con 0x7fb1101097b0 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f80035b0 con 0x7fb1101097b0 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb104001830 con 0x7fb1101097b0 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3484999072 0 0) 0x7fb1101a7af0 con 0x7fb11010d380 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 >> v1:172.21.15.119:6789/0 conn(0x7fb1101097b0 legacy=0x7fb1101a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 >> v1:172.21.15.114:6789/0 conn(0x7fb110106900 legacy=0x7fb11010cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1101a9ef0 con 0x7fb11010d380 2024-03-24T11:39:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1101a6b00 con 0x7fb11010d380 2024-03-24T11:39:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1101a6f80 con 0x7fb11010d380 2024-03-24T11:39:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fb108003eb0 con 0x7fb11010d380 2024-03-24T11:39:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.078+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb108004ce0 con 0x7fb11010d380 2024-03-24T11:39:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.079+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f0005120 con 0x7fb11010d380 2024-03-24T11:39:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.080+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 17) v1 ==== 91111+0+0 (unknown 3770792243 0 0) 0x7fb108003a20 con 0x7fb11010d380 2024-03-24T11:39:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.080+0000 7fb10e59c700 1 -- 172.21.15.114:0/3951120 >> v1:172.21.15.114:6800/3103139641 conn(0x7fb0f806e900 legacy=0x7fb0f8070dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/3103139641 2024-03-24T11:39:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.080+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (unknown 2262200551 0 0) 0x7fb108087380 con 0x7fb11010d380 2024-03-24T11:39:59.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.084+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fb108050c80 con 0x7fb11010d380 2024-03-24T11:39:59.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.281+0000 7fb10e59c700 1 -- 172.21.15.114:0/3951120 >> v1:172.21.15.114:6800/3103139641 conn(0x7fb0f806e900 legacy=0x7fb0f8070dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.114:6800/3103139641 2024-03-24T11:39:59.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.339+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb0f00053d0 con 0x7fb11010d380 2024-03-24T11:39:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.342+0000 7fb0ff7fe700 1 -- 172.21.15.114:0/3951120 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+174272 (unknown 831476310 0 748951477) 0x7fb108052bb0 con 0x7fb11010d380 2024-03-24T11:39:59.345 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:39:59.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.345+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 >> v1:172.21.15.114:6800/3103139641 conn(0x7fb0f806e900 legacy=0x7fb0f8070dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:39:59.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.345+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 >> v1:172.21.15.114:6790/0 conn(0x7fb11010d380 legacy=0x7fb1101a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:39:59.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.345+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 shutdown_connections 2024-03-24T11:39:59.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.345+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 >> 172.21.15.114:0/3951120 conn(0x7fb1100fc410 msgr2=0x7fb110069b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:39:59.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.345+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 shutdown_connections 2024-03-24T11:39:59.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:39:59.345+0000 7fb114fae700 1 -- 172.21.15.114:0/3951120 wait complete. 2024-03-24T11:39:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:39:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3951120' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:39:59.554 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:39:59 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:39:59.496+0000 7f9ea6feb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:39:59.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:39:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3951120' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:39:59.725 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:39:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3951120' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:39:59.725 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:39:59 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:39:59.448+0000 7f38682ea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:40:00.178 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6800","nonce":3103139641}]},"active_addr":"172.21.15.114:6800/3103139641","active_change":"2024-03-24T11:34:46.998014+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4287975194}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1527504117}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":587225835}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4202635877}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3282312985}]}]} 2024-03-24T11:40:00.182 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:40:00.182 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:40:00.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:40:00.354 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:40:00.024+0000 7f38682ea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:40:00.354 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:40:00.125+0000 7f38682ea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:40:00.354 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:40:00.353+0000 7f38682ea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:40:00.373 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:40:00.057+0000 7f9ea6feb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:40:00.373 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:40:00.154+0000 7f9ea6feb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:40:00.487 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:00] ENGINE Bus STARTING 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: CherryPy Checker: 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: The Application mounted at '' has an empty config. 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:00] ENGINE Serving on http://:::9283 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:00] ENGINE Bus STARTED 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: overall HEALTH_OK 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: Active manager daemon y restarted 2024-03-24T11:40:00.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: Activating manager daemon y 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: mgrmap e18: y(active, starting, since 0.0116958s), standbys: x 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:40:00.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: Standby manager daemon x restarted 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: Standby manager daemon x started 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[30312]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: overall HEALTH_OK 2024-03-24T11:40:00.615 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: Active manager daemon y restarted 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: Activating manager daemon y 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: mgrmap e18: y(active, starting, since 0.0116958s), standbys: x 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:40:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: Standby manager daemon x restarted 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: Standby manager daemon x started 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:40:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:00 smithi114 ceph-mon[42466]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:40:00.371+0000 7f9ea6feb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: [24/Mar/2024:11:40:00] ENGINE Bus STARTING 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: CherryPy Checker: 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: The Application mounted at '' has an empty config. 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: [24/Mar/2024:11:40:00] ENGINE Serving on http://:::9283 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x[35787]: [24/Mar/2024:11:40:00] ENGINE Bus STARTED 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: overall HEALTH_OK 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: Active manager daemon y restarted 2024-03-24T11:40:00.624 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: Activating manager daemon y 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: mgrmap e18: y(active, starting, since 0.0116958s), standbys: x 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:40:00.625 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: Standby manager daemon x restarted 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: Standby manager daemon x started 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:40:00.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:00 smithi119 ceph-mon[33714]: from='mgr.? v1:172.21.15.119:0/2620436297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:40:01.642 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: Manager daemon y is now available 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:40:01.643 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:01 smithi119 ceph-mon[33714]: mgrmap e19: y(active, since 1.01716s), standbys: x 2024-03-24T11:40:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: Manager daemon y is now available 2024-03-24T11:40:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[30312]: mgrmap e19: y(active, since 1.01716s), standbys: x 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: Manager daemon y is now available 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:40:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:01 smithi114 ceph-mon[42466]: mgrmap e19: y(active, since 1.01716s), standbys: x 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.532+0000 7ff39362d700 1 -- 172.21.15.114:0/1868526986 >> v1:172.21.15.114:6790/0 conn(0x7ff38c106900 legacy=0x7ff38c106ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- 172.21.15.114:0/1868526986 shutdown_connections 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- 172.21.15.114:0/1868526986 >> 172.21.15.114:0/1868526986 conn(0x7ff38c0fc410 msgr2=0x7ff38c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- 172.21.15.114:0/1868526986 shutdown_connections 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- 172.21.15.114:0/1868526986 wait complete. 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 Processor -- start 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- start start 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff38c1a67b0 con 0x7ff38c10d380 2024-03-24T11:40:02.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff38c1a79b0 con 0x7ff38c1097b0 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.533+0000 7ff39362d700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff38c1a8bb0 con 0x7ff38c106900 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff391bca700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff38c10d380 0x7ff38c1a4e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:45358/0 (socket says 172.21.15.114:45358) 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff391bca700 1 -- 172.21.15.114:0/4085728671 learned_addr learned my addr 172.21.15.114:0/4085728671 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3913c9700 1 --1- 172.21.15.114:0/4085728671 >> v1:172.21.15.114:6790/0 conn(0x7ff38c106900 0x7ff38c10c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:40980/0 (socket says 172.21.15.114:40980) 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308359084 0 0) 0x7ff38c1a67b0 con 0x7ff38c10d380 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3780035b0 con 0x7ff38c10d380 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133013195 0 0) 0x7ff38c1a8bb0 con 0x7ff38c106900 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c1a67b0 con 0x7ff38c106900 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648281241 0 0) 0x7ff38c1a79b0 con 0x7ff38c1097b0 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c1a8bb0 con 0x7ff38c1097b0 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1380188049 0 0) 0x7ff3780035b0 con 0x7ff38c10d380 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff38c1a79b0 con 0x7ff38c10d380 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff384001d50 con 0x7ff38c10d380 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244580933 0 0) 0x7ff38c1a67b0 con 0x7ff38c106900 2024-03-24T11:40:02.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3780035b0 con 0x7ff38c106900 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff388004460 con 0x7ff38c106900 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243619789 0 0) 0x7ff38c1a8bb0 con 0x7ff38c1097b0 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff38c1a67b0 con 0x7ff38c1097b0 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2929484627 0 0) 0x7ff38c1a79b0 con 0x7ff38c10d380 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 >> v1:172.21.15.114:6790/0 conn(0x7ff38c106900 legacy=0x7ff38c10c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 >> v1:172.21.15.119:6789/0 conn(0x7ff38c1097b0 legacy=0x7ff38c1a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c1a9db0 con 0x7ff38c10d380 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.534+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c1a8da0 con 0x7ff38c10d380 2024-03-24T11:40:02.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.535+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c1a9220 con 0x7ff38c10d380 2024-03-24T11:40:02.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.535+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ff384003e90 con 0x7ff38c10d380 2024-03-24T11:40:02.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.535+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff384005f60 con 0x7ff38c10d380 2024-03-24T11:40:02.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.536+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff370005120 con 0x7ff38c10d380 2024-03-24T11:40:02.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.536+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 19) v1 ==== 90854+0+0 (unknown 1100578042 0 0) 0x7ff38401c470 con 0x7ff38c10d380 2024-03-24T11:40:02.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.537+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7ff384088060 con 0x7ff38c10d380 2024-03-24T11:40:02.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.541+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ff384052ab0 con 0x7ff38c10d380 2024-03-24T11:40:02.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[30312]: [24/Mar/2024:11:40:00] ENGINE Bus STARTING 2024-03-24T11:40:02.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[30312]: [24/Mar/2024:11:40:00] ENGINE Serving on https://172.21.15.114:7150 2024-03-24T11:40:02.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[30312]: [24/Mar/2024:11:40:01] ENGINE Serving on http://172.21.15.114:8765 2024-03-24T11:40:02.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[30312]: [24/Mar/2024:11:40:01] ENGINE Bus STARTED 2024-03-24T11:40:02.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:02.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[30312]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:02.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[42466]: [24/Mar/2024:11:40:00] ENGINE Bus STARTING 2024-03-24T11:40:02.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[42466]: [24/Mar/2024:11:40:00] ENGINE Serving on https://172.21.15.114:7150 2024-03-24T11:40:02.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[42466]: [24/Mar/2024:11:40:01] ENGINE Serving on http://172.21.15.114:8765 2024-03-24T11:40:02.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[42466]: [24/Mar/2024:11:40:01] ENGINE Bus STARTED 2024-03-24T11:40:02.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:02.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:02 smithi114 ceph-mon[42466]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:02.702 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:02 smithi119 ceph-mon[33714]: [24/Mar/2024:11:40:00] ENGINE Bus STARTING 2024-03-24T11:40:02.702 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:02 smithi119 ceph-mon[33714]: [24/Mar/2024:11:40:00] ENGINE Serving on https://172.21.15.114:7150 2024-03-24T11:40:02.702 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:02 smithi119 ceph-mon[33714]: [24/Mar/2024:11:40:01] ENGINE Serving on http://172.21.15.114:8765 2024-03-24T11:40:02.702 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:02 smithi119 ceph-mon[33714]: [24/Mar/2024:11:40:01] ENGINE Bus STARTED 2024-03-24T11:40:02.702 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:02 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:02.702 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:02 smithi119 ceph-mon[33714]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:02.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.746+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3700053d0 con 0x7ff38c10d380 2024-03-24T11:40:02.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.747+0000 7ff3827fc700 1 -- 172.21.15.114:0/4085728671 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+21088 (unknown 1686360691 0 1754737836) 0x7ff384054bb0 con 0x7ff38c10d380 2024-03-24T11:40:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:40:02.748 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":72,"fsid":"412c996a-e9d2-11ee-95cd-87774f69a715","created":"2024-03-24T11:33:52.997920+0000","modified":"2024-03-24T11:40:00.366744+0000","last_up_change":"2024-03-24T11:39:08.723989+0000","last_in_change":"2024-03-24T11:38:50.789246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:37:03.321298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T11:39:20.738412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T11:39:21.252983+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:39:22.576003+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:39:24.568627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:39:26.642943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f7dfcce-3dae-4acc-b2b7-849ea3e22f73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6801","nonce":29788729}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6802","nonce":29788729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6804","nonce":29788729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6803","nonce":29788729}]},"public_addr":"172.21.15.114:6801/29788729","cluster_addr":"172.21.15.114:6802/29788729","heartbeat_back_addr":"172.21.15.114:6804/29788729","heartbeat_front_addr":"172.21.15.114:6803/29788729","state":["exists","up"]},{"osd":1,"uuid":"2439ca79-43cf-42d7-95ad-1540a558405e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6805","nonce":1920152174}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6806","nonce":1920152174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6808","nonce":1920152174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6807","nonce":1920152174}]},"public_addr":"172.21.15.114:6805/1920152174","cluster_addr":"172.21.15.114:6806/1920152174","heartbeat_back_addr":"172.21.15.114:6808/1920152174","heartbeat_front_addr":"172.21.15.114:6807/1920152174","state":["exists","up"]},{"osd":2,"uuid":"1f1a5efc-c70d-4103-bb5b-d886f0d38540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6809","nonce":2539296009}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6810","nonce":2539296009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6812","nonce":2539296009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6811","nonce":2539296009}]},"public_addr":"172.21.15.114:6809/2539296009","cluster_addr":"172.21.15.114:6810/2539296009","heartbeat_back_addr":"172.21.15.114:6812/2539296009","heartbeat_front_addr":"172.21.15.114:6811/2539296009","state":["exists","up"]},{"osd":3,"uuid":"34da3c83-66b6-45f3-bf20-6b7fc06e20e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6813","nonce":3141649135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6814","nonce":3141649135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6816","nonce":3141649135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6815","nonce":3141649135}]},"public_addr":"172.21.15.114:6813/3141649135","cluster_addr":"172.21.15.114:6814/3141649135","heartbeat_back_addr":"172.21.15.114:6816/3141649135","heartbeat_front_addr":"172.21.15.114:6815/3141649135","state":["exists","up"]},{"osd":4,"uuid":"305f87ac-0068-49dd-8b7b-24909140de3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6800","nonce":1375520258}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6801","nonce":1375520258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6803","nonce":1375520258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6802","nonce":1375520258}]},"public_addr":"172.21.15.119:6800/1375520258","cluster_addr":"172.21.15.119:6801/1375520258","heartbeat_back_addr":"172.21.15.119:6803/1375520258","heartbeat_front_addr":"172.21.15.119:6802/1375520258","state":["exists","up"]},{"osd":5,"uuid":"9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6804","nonce":4116082199}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6805","nonce":4116082199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6807","nonce":4116082199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6806","nonce":4116082199}]},"public_addr":"172.21.15.119:6804/4116082199","cluster_addr":"172.21.15.119:6805/4116082199","heartbeat_back_addr":"172.21.15.119:6807/4116082199","heartbeat_front_addr":"172.21.15.119:6806/4116082199","state":["exists","up"]},{"osd":6,"uuid":"5ef919b8-522c-4a53-b548-49a61d64ccce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6808","nonce":1563073715}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6809","nonce":1563073715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6811","nonce":1563073715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6810","nonce":1563073715}]},"public_addr":"172.21.15.119:6808/1563073715","cluster_addr":"172.21.15.119:6809/1563073715","heartbeat_back_addr":"172.21.15.119:6811/1563073715","heartbeat_front_addr":"172.21.15.119:6810/1563073715","state":["exists","up"]},{"osd":7,"uuid":"46dea5e4-e52f-43f0-8069-851bdb0d7e62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6812","nonce":2022246932}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6813","nonce":2022246932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6815","nonce":2022246932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6814","nonce":2022246932}]},"public_addr":"172.21.15.119:6812/2022246932","cluster_addr":"172.21.15.119:6813/2022246932","heartbeat_back_addr":"172.21.15.119:6815/2022246932","heartbeat_front_addr":"172.21.15.119:6814/2022246932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:09.396337+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:34.210168+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:59.825426+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:37:25.558654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:37:49.673911+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:38:14.826924+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:38:40.981951+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:39:06.959944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1527504117":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/4202635877":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/3282312985":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/3331501023":"2024-03-25T11:34:46.997878+0000","172.21.15.114:0/4287975194":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/3312362946":"2024-03-25T11:34:46.997878+0000","172.21.15.114:6800/3103139641":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/309866353":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/587225835":"2024-03-25T11:40:00.366711+0000","172.21.15.114:6800/1606950246":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/1453553075":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/319492564":"2024-03-25T11:34:20.323788+0000","172.21.15.114:6800/773139778":"2024-03-25T11:34:46.997878+0000","172.21.15.114:0/581628855":"2024-03-25T11:34:46.997878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:40:02.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.749+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 >> v1:172.21.15.114:6800/1253397276 conn(0x7ff37806e410 legacy=0x7ff3780708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:02.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.749+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 >> v1:172.21.15.114:6789/0 conn(0x7ff38c10d380 legacy=0x7ff38c1a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:02.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.750+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 shutdown_connections 2024-03-24T11:40:02.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.750+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 >> 172.21.15.114:0/4085728671 conn(0x7ff38c0fc410 msgr2=0x7ff38c110010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:02.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.750+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 shutdown_connections 2024-03-24T11:40:02.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:02.750+0000 7ff39362d700 1 -- 172.21.15.114:0/4085728671 wait complete. 2024-03-24T11:40:03.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:03 smithi114 ceph-mon[30312]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:03.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4085728671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:40:03.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:03 smithi114 ceph-mon[30312]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-24T11:40:03.713 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:03 smithi114 ceph-mon[42466]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:03.714 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4085728671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:40:03.714 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:03 smithi114 ceph-mon[42466]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-24T11:40:03.748 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:40:03.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:40:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:03 smithi119 ceph-mon[33714]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4085728671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:40:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:03 smithi119 ceph-mon[33714]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-24T11:40:04.011 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:04.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:40:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:05.187 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[30312]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.188 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:05 smithi114 ceph-mon[42466]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:05.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:05.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:05 smithi119 ceph-mon[33714]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.702+0000 7ff9f655f700 1 -- 172.21.15.114:0/3752194743 >> v1:172.21.15.114:6790/0 conn(0x7ff9f01097b0 legacy=0x7ff9f010bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.702+0000 7ff9f655f700 1 -- 172.21.15.114:0/3752194743 shutdown_connections 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.702+0000 7ff9f655f700 1 -- 172.21.15.114:0/3752194743 >> 172.21.15.114:0/3752194743 conn(0x7ff9f00fc410 msgr2=0x7ff9f00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.702+0000 7ff9f655f700 1 -- 172.21.15.114:0/3752194743 shutdown_connections 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.702+0000 7ff9f655f700 1 -- 172.21.15.114:0/3752194743 wait complete. 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.703+0000 7ff9f655f700 1 Processor -- start 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.703+0000 7ff9f655f700 1 -- start start 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.703+0000 7ff9f655f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9f01a6930 con 0x7ff9f010d380 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.703+0000 7ff9f655f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9f01a7b10 con 0x7ff9f01097b0 2024-03-24T11:40:05.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.703+0000 7ff9f655f700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9f01a8d10 con 0x7ff9f0106900 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.703+0000 7ff9ef7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff9f01097b0 0x7ff9f01a1740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.114:39530/0 (socket says 172.21.15.114:39530) 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.703+0000 7ff9ef7fe700 1 -- 172.21.15.114:0/580014879 learned_addr learned my addr 172.21.15.114:0/580014879 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286949251 0 0) 0x7ff9f01a7b10 con 0x7ff9f01097b0 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d80035b0 con 0x7ff9f01097b0 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233541016 0 0) 0x7ff9f01a6930 con 0x7ff9f010d380 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f01a7b10 con 0x7ff9f010d380 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631987116 0 0) 0x7ff9f01a8d10 con 0x7ff9f0106900 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f01a6930 con 0x7ff9f0106900 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242093610 0 0) 0x7ff9d80035b0 con 0x7ff9f01097b0 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f01a8d10 con 0x7ff9f01097b0 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff9e4001370 con 0x7ff9f01097b0 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2454999655 0 0) 0x7ff9f01a7b10 con 0x7ff9f010d380 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9d80035b0 con 0x7ff9f010d380 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff9e8001c30 con 0x7ff9f010d380 2024-03-24T11:40:05.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1047955486 0 0) 0x7ff9f01a6930 con 0x7ff9f0106900 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f01a7b10 con 0x7ff9f0106900 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff9e0001630 con 0x7ff9f0106900 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 396608686 0 0) 0x7ff9f01a8d10 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 >> v1:172.21.15.114:6790/0 conn(0x7ff9f0106900 legacy=0x7ff9f010cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 >> v1:172.21.15.114:6789/0 conn(0x7ff9f010d380 legacy=0x7ff9f01a5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f01a9f10 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f01a6b20 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f01a6fa0 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.704+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ff9e4001ab0 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.705+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff9e4004e60 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.705+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9dc005120 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.706+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7ff9e40037f0 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.706+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7ff9e4088450 con 0x7ff9f01097b0 2024-03-24T11:40:05.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.711+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ff9e4055930 con 0x7ff9f01097b0 2024-03-24T11:40:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.922+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9dc0053d0 con 0x7ff9f01097b0 2024-03-24T11:40:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.924+0000 7ff9ed7fa700 1 -- 172.21.15.114:0/580014879 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+21088 (unknown 1686360691 0 1754737836) 0x7ff9e4053ca0 con 0x7ff9f01097b0 2024-03-24T11:40:05.925 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:40:05.925 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":72,"fsid":"412c996a-e9d2-11ee-95cd-87774f69a715","created":"2024-03-24T11:33:52.997920+0000","modified":"2024-03-24T11:40:00.366744+0000","last_up_change":"2024-03-24T11:39:08.723989+0000","last_in_change":"2024-03-24T11:38:50.789246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:37:03.321298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T11:39:20.738412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T11:39:21.252983+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:39:22.576003+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:39:24.568627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:39:26.642943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f7dfcce-3dae-4acc-b2b7-849ea3e22f73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6801","nonce":29788729}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6802","nonce":29788729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6804","nonce":29788729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6803","nonce":29788729}]},"public_addr":"172.21.15.114:6801/29788729","cluster_addr":"172.21.15.114:6802/29788729","heartbeat_back_addr":"172.21.15.114:6804/29788729","heartbeat_front_addr":"172.21.15.114:6803/29788729","state":["exists","up"]},{"osd":1,"uuid":"2439ca79-43cf-42d7-95ad-1540a558405e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6805","nonce":1920152174}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6806","nonce":1920152174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6808","nonce":1920152174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6807","nonce":1920152174}]},"public_addr":"172.21.15.114:6805/1920152174","cluster_addr":"172.21.15.114:6806/1920152174","heartbeat_back_addr":"172.21.15.114:6808/1920152174","heartbeat_front_addr":"172.21.15.114:6807/1920152174","state":["exists","up"]},{"osd":2,"uuid":"1f1a5efc-c70d-4103-bb5b-d886f0d38540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6809","nonce":2539296009}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6810","nonce":2539296009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6812","nonce":2539296009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6811","nonce":2539296009}]},"public_addr":"172.21.15.114:6809/2539296009","cluster_addr":"172.21.15.114:6810/2539296009","heartbeat_back_addr":"172.21.15.114:6812/2539296009","heartbeat_front_addr":"172.21.15.114:6811/2539296009","state":["exists","up"]},{"osd":3,"uuid":"34da3c83-66b6-45f3-bf20-6b7fc06e20e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6813","nonce":3141649135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6814","nonce":3141649135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6816","nonce":3141649135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.114:6815","nonce":3141649135}]},"public_addr":"172.21.15.114:6813/3141649135","cluster_addr":"172.21.15.114:6814/3141649135","heartbeat_back_addr":"172.21.15.114:6816/3141649135","heartbeat_front_addr":"172.21.15.114:6815/3141649135","state":["exists","up"]},{"osd":4,"uuid":"305f87ac-0068-49dd-8b7b-24909140de3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6800","nonce":1375520258}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6801","nonce":1375520258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6803","nonce":1375520258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6802","nonce":1375520258}]},"public_addr":"172.21.15.119:6800/1375520258","cluster_addr":"172.21.15.119:6801/1375520258","heartbeat_back_addr":"172.21.15.119:6803/1375520258","heartbeat_front_addr":"172.21.15.119:6802/1375520258","state":["exists","up"]},{"osd":5,"uuid":"9d0e9a5e-ce62-45b7-b29d-99aaf65f50b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6804","nonce":4116082199}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6805","nonce":4116082199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6807","nonce":4116082199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6806","nonce":4116082199}]},"public_addr":"172.21.15.119:6804/4116082199","cluster_addr":"172.21.15.119:6805/4116082199","heartbeat_back_addr":"172.21.15.119:6807/4116082199","heartbeat_front_addr":"172.21.15.119:6806/4116082199","state":["exists","up"]},{"osd":6,"uuid":"5ef919b8-522c-4a53-b548-49a61d64ccce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6808","nonce":1563073715}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6809","nonce":1563073715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6811","nonce":1563073715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6810","nonce":1563073715}]},"public_addr":"172.21.15.119:6808/1563073715","cluster_addr":"172.21.15.119:6809/1563073715","heartbeat_back_addr":"172.21.15.119:6811/1563073715","heartbeat_front_addr":"172.21.15.119:6810/1563073715","state":["exists","up"]},{"osd":7,"uuid":"46dea5e4-e52f-43f0-8069-851bdb0d7e62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6812","nonce":2022246932}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6813","nonce":2022246932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6815","nonce":2022246932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.119:6814","nonce":2022246932}]},"public_addr":"172.21.15.119:6812/2022246932","cluster_addr":"172.21.15.119:6813/2022246932","heartbeat_back_addr":"172.21.15.119:6815/2022246932","heartbeat_front_addr":"172.21.15.119:6814/2022246932","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:09.396337+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:34.210168+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:36:59.825426+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:37:25.558654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:37:49.673911+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:38:14.826924+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:38:40.981951+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:39:06.959944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1527504117":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/4202635877":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/3282312985":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/3331501023":"2024-03-25T11:34:46.997878+0000","172.21.15.114:0/4287975194":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/3312362946":"2024-03-25T11:34:46.997878+0000","172.21.15.114:6800/3103139641":"2024-03-25T11:40:00.366711+0000","172.21.15.114:0/309866353":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/587225835":"2024-03-25T11:40:00.366711+0000","172.21.15.114:6800/1606950246":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/1453553075":"2024-03-25T11:34:20.323788+0000","172.21.15.114:0/319492564":"2024-03-25T11:34:20.323788+0000","172.21.15.114:6800/773139778":"2024-03-25T11:34:46.997878+0000","172.21.15.114:0/581628855":"2024-03-25T11:34:46.997878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:40:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.926+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 >> v1:172.21.15.114:6800/1253397276 conn(0x7ff9d806e940 legacy=0x7ff9d8070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.926+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 >> v1:172.21.15.119:6789/0 conn(0x7ff9f01097b0 legacy=0x7ff9f01a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.926+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 shutdown_connections 2024-03-24T11:40:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.926+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 >> 172.21.15.114:0/580014879 conn(0x7ff9f00fc410 msgr2=0x7ff9f0069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.926+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 shutdown_connections 2024-03-24T11:40:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:05.926+0000 7ff9f655f700 1 -- 172.21.15.114:0/580014879 wait complete. 2024-03-24T11:40:06.310 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.310 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.310 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.310 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi119 to 3760M 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:06.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/580014879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi119 to 3760M 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:06.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:40:06.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/580014879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:40:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi119 to 3760M 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:40:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/580014879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:40:06.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:40:06.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:40:06.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:40:06.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T11:40:06.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T11:40:06.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T11:40:06.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T11:40:06.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T11:40:07.181 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.236 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.269 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.275 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.276 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.519 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:07.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi114 to 3760M 2024-03-24T11:40:07.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-24T11:40:07.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:40:07.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[30312]: Deploying daemon alertmanager.a on smithi114 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi114 to 3760M 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:07 smithi114 ceph-mon[42466]: Deploying daemon alertmanager.a on smithi114 2024-03-24T11:40:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi114 to 3760M 2024-03-24T11:40:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-24T11:40:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Updating smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:07 smithi119 ceph-mon[33714]: Deploying daemon alertmanager.a on smithi114 2024-03-24T11:40:09.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:09 smithi114 ceph-mon[30312]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:40:09.602 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:09 smithi114 ceph-mon[42466]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:40:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:09 smithi119 ceph-mon[33714]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.190+0000 7fe916944700 1 -- 172.21.15.114:0/1603002805 >> v1:172.21.15.114:6789/0 conn(0x7fe91010e490 legacy=0x7fe910110950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.190+0000 7fe916944700 1 -- 172.21.15.114:0/1603002805 shutdown_connections 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.190+0000 7fe916944700 1 -- 172.21.15.114:0/1603002805 >> 172.21.15.114:0/1603002805 conn(0x7fe9100fd520 msgr2=0x7fe9100ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.190+0000 7fe916944700 1 -- 172.21.15.114:0/1603002805 shutdown_connections 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.190+0000 7fe916944700 1 -- 172.21.15.114:0/1603002805 wait complete. 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.190+0000 7fe916944700 1 Processor -- start 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.190+0000 7fe916944700 1 -- start start 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe916944700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe9101a8b10 con 0x7fe91010a8c0 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe916944700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe9101a9cb0 con 0x7fe910107a10 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe916944700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe9101aaeb0 con 0x7fe91010e490 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90f7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe91010a8c0 0x7fe9101a3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55746/0 (socket says 172.21.15.114:55746) 2024-03-24T11:40:11.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90f7fe700 1 -- 172.21.15.114:0/2568308196 learned_addr learned my addr 172.21.15.114:0/2568308196 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968515287 0 0) 0x7fe9101a8b10 con 0x7fe91010a8c0 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f80035b0 con 0x7fe91010a8c0 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663023211 0 0) 0x7fe9101a9cb0 con 0x7fe910107a10 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9101a8b10 con 0x7fe910107a10 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988240202 0 0) 0x7fe9101aaeb0 con 0x7fe91010e490 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.191+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9101a9cb0 con 0x7fe91010e490 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143250153 0 0) 0x7fe8f80035b0 con 0x7fe91010a8c0 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9101aaeb0 con 0x7fe91010a8c0 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145437067 0 0) 0x7fe9101a8b10 con 0x7fe910107a10 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8f80035b0 con 0x7fe910107a10 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe904001970 con 0x7fe91010a8c0 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235447519 0 0) 0x7fe9101a9cb0 con 0x7fe91010e490 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9101a8b10 con 0x7fe91010e490 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe900001830 con 0x7fe910107a10 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe9080046a0 con 0x7fe91010e490 2024-03-24T11:40:11.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4011144279 0 0) 0x7fe9101aaeb0 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 >> v1:172.21.15.114:6790/0 conn(0x7fe91010e490 legacy=0x7fe9101a7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 >> v1:172.21.15.119:6789/0 conn(0x7fe910107a10 legacy=0x7fe910075850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9101ac0b0 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9101a9ea0 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9101aa320 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fe904001b10 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.192+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fe904005bd0 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.193+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.114:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe910112420 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.194+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7fe90401c120 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.194+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7fe9040883e0 con 0x7fe91010a8c0 2024-03-24T11:40:11.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.194+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.119:6804/4116082199 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8f807cc10 con 0x7fe8f8078f00 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.194+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7fe9040887f0 con 0x7fe91010a8c0 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.196+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== osd.5 v1:172.21.15.119:6804/4116082199 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe8f807cc10 con 0x7fe8f8078f00 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.213+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 --> v1:172.21.15.119:6804/4116082199 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9101aa860 con 0x7fe8f8078f00 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.214+0000 7fe90d7fa700 1 -- 172.21.15.114:0/2568308196 <== osd.5 v1:172.21.15.119:6804/4116082199 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 398800033) 0x7fe9101aa860 con 0x7fe8f8078f00 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.214+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 >> v1:172.21.15.119:6804/4116082199 conn(0x7fe8f8078f00 legacy=0x7fe8f807b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.215+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 >> v1:172.21.15.114:6800/1253397276 conn(0x7fe8f806e790 legacy=0x7fe8f8070c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.215+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 >> v1:172.21.15.114:6789/0 conn(0x7fe91010a8c0 legacy=0x7fe9101a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.215+0000 7fe914ee1700 1 -- 172.21.15.114:0/2568308196 reap_dead start 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.215+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 shutdown_connections 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.215+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 >> 172.21.15.114:0/2568308196 conn(0x7fe9100fd520 msgr2=0x7fe910101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.215+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 shutdown_connections 2024-03-24T11:40:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:11.215+0000 7fe916944700 1 -- 172.21.15.114:0/2568308196 wait complete. 2024-03-24T11:40:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:11 smithi114 ceph-mon[30312]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:40:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:11 smithi114 ceph-mon[42466]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:40:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:11 smithi119 ceph-mon[33714]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:40:11.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:40:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:40:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:12 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:12 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:12 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:13 smithi114 ceph-mon[30312]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:40:13.666 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:13 smithi114 ceph-mon[42466]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:40:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:13 smithi119 ceph-mon[33714]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:40:14.891 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:40:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:15.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:15 smithi114 ceph-mon[30312]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:40:15.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:15 smithi114 ceph-mon[42466]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:40:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:15 smithi119 ceph-mon[33714]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:40:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:16 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:16.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:16 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:16 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:17 smithi114 ceph-mon[30312]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:40:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:17 smithi114 ceph-mon[42466]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:40:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:17 smithi119 ceph-mon[33714]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:40:18.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:18 smithi114 ceph-mon[42466]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:18 smithi114 ceph-mon[30312]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:18 smithi119 ceph-mon[33714]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.813+0000 7f493ae9e700 1 -- 172.21.15.114:0/133638472 >> v1:172.21.15.114:6790/0 conn(0x7f493410aa70 legacy=0x7f493410ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.813+0000 7f493ae9e700 1 -- 172.21.15.114:0/133638472 shutdown_connections 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.813+0000 7f493ae9e700 1 -- 172.21.15.114:0/133638472 >> 172.21.15.114:0/133638472 conn(0x7f49340fc410 msgr2=0x7f49340fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.813+0000 7f493ae9e700 1 -- 172.21.15.114:0/133638472 shutdown_connections 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.813+0000 7f493ae9e700 1 -- 172.21.15.114:0/133638472 wait complete. 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.814+0000 7f493ae9e700 1 Processor -- start 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.814+0000 7f493ae9e700 1 -- start start 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.814+0000 7f493ae9e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f49341a8ad0 con 0x7f4934102c60 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.814+0000 7f493ae9e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f49341a9c70 con 0x7f493410aa70 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.814+0000 7f493ae9e700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f49341aae70 con 0x7f493410e580 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4938c3a700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4934102c60 0x7f493410a0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38600/0 (socket says 172.21.15.114:38600) 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4938c3a700 1 -- 172.21.15.114:0/390844778 learned_addr learned my addr 172.21.15.114:0/390844778 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907652128 0 0) 0x7f49341a8ad0 con 0x7f4934102c60 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49240035b0 con 0x7f4934102c60 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785121262 0 0) 0x7f49341aae70 con 0x7f493410e580 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49341a8ad0 con 0x7f493410e580 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148351624 0 0) 0x7f49341a8ad0 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49341aae70 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3592377094 0 0) 0x7f49240035b0 con 0x7f4934102c60 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49341a8ad0 con 0x7f4934102c60 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f492c001c30 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f49280012d0 con 0x7f4934102c60 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100154009 0 0) 0x7f49341a9c70 con 0x7f493410aa70 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49240035b0 con 0x7f493410aa70 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2863305125 0 0) 0x7f49341aae70 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 >> v1:172.21.15.119:6789/0 conn(0x7f493410aa70 legacy=0x7f49341a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 >> v1:172.21.15.114:6789/0 conn(0x7f4934102c60 legacy=0x7f493410a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49341ac070 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.815+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49341ab060 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.816+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49341ab4e0 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.816+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f492c003eb0 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.816+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f492c004da0 con 0x7f493410e580 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.817+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4918000f80 con 0x7f493410e580 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.817+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f492c005fb0 con 0x7f493410e580 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.817+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f492c0873f0 con 0x7f493410e580 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.818+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6801/29788729 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f492407cbe0 con 0x7f4924078ed0 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.818+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f492c087820 con 0x7f493410e580 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.819+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== osd.0 v1:172.21.15.114:6801/29788729 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f492407cbe0 con 0x7f4924078ed0 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 --> v1:172.21.15.114:6801/29788729 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4918002c40 con 0x7f4924078ed0 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f4931ffb700 1 -- 172.21.15.114:0/390844778 <== osd.0 v1:172.21.15.114:6801/29788729 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2240242322) 0x7f4918002c40 con 0x7f4924078ed0 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 >> v1:172.21.15.114:6801/29788729 conn(0x7f4924078ed0 legacy=0x7f492407b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 >> v1:172.21.15.114:6800/1253397276 conn(0x7f492406e5e0 legacy=0x7f4924070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 >> v1:172.21.15.114:6790/0 conn(0x7f493410e580 legacy=0x7f49341a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493943b700 1 -- 172.21.15.114:0/390844778 reap_dead start 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 shutdown_connections 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 >> 172.21.15.114:0/390844778 conn(0x7f49340fc410 msgr2=0x7f49340fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 shutdown_connections 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.835+0000 7f493ae9e700 1 -- 172.21.15.114:0/390844778 wait complete. 2024-03-24T11:40:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:21 smithi114 ceph-mon[30312]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:21 smithi114 ceph-mon[42466]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:21 smithi119 ceph-mon[33714]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:21.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:40:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:40:22.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- 172.21.15.114:0/3713481728 >> v1:172.21.15.114:6789/0 conn(0x7f9258106910 legacy=0x7f9258106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:22.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- 172.21.15.114:0/3713481728 shutdown_connections 2024-03-24T11:40:22.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- 172.21.15.114:0/3713481728 >> 172.21.15.114:0/3713481728 conn(0x7f92580fc420 msgr2=0x7f92580fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:22.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- 172.21.15.114:0/3713481728 shutdown_connections 2024-03-24T11:40:22.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- 172.21.15.114:0/3713481728 wait complete. 2024-03-24T11:40:22.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 Processor -- start 2024-03-24T11:40:22.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- start start 2024-03-24T11:40:22.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92581a6890 con 0x7f925810d390 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92581a7a70 con 0x7f9258106910 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.327+0000 7f925eb6f700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f92581a8c70 con 0x7f92581097c0 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9257fff700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f92581097c0 0x7f92581a1750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:44680/0 (socket says 172.21.15.114:44680) 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9257fff700 1 -- 172.21.15.114:0/465421797 learned_addr learned my addr 172.21.15.114:0/465421797 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f925d10c700 1 --1- 172.21.15.114:0/465421797 >> v1:172.21.15.114:6789/0 conn(0x7f925810d390 0x7f92581a4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38622/0 (socket says 172.21.15.114:38622) 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277002543 0 0) 0x7f92581a6890 con 0x7f925810d390 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92480049e0 con 0x7f925810d390 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340278136 0 0) 0x7f92581a8c70 con 0x7f92581097c0 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92581a6890 con 0x7f92581097c0 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797480654 0 0) 0x7f92581a7a70 con 0x7f9258106910 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92581a8c70 con 0x7f9258106910 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2502855737 0 0) 0x7f92480049e0 con 0x7f925810d390 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92581a7a70 con 0x7f925810d390 2024-03-24T11:40:22.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4121643442 0 0) 0x7f92581a6890 con 0x7f92581097c0 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92480049e0 con 0x7f92581097c0 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9250001430 con 0x7f925810d390 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3667343138 0 0) 0x7f92581a8c70 con 0x7f9258106910 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92581a6890 con 0x7f9258106910 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f92440017f0 con 0x7f92581097c0 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f924c0019c0 con 0x7f9258106910 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 746163020 0 0) 0x7f92581a7a70 con 0x7f925810d390 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 >> v1:172.21.15.114:6790/0 conn(0x7f92581097c0 legacy=0x7f92581a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 >> v1:172.21.15.119:6789/0 conn(0x7f9258106910 legacy=0x7f925810ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92581a9e70 con 0x7f925810d390 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.328+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92581a8e60 con 0x7f925810d390 2024-03-24T11:40:22.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.329+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f9250001c60 con 0x7f925810d390 2024-03-24T11:40:22.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.329+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9250005df0 con 0x7f925810d390 2024-03-24T11:40:22.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.329+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92581a92e0 con 0x7f925810d390 2024-03-24T11:40:22.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.329+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9258113720 con 0x7f925810d390 2024-03-24T11:40:22.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.330+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f9250003a00 con 0x7f925810d390 2024-03-24T11:40:22.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.330+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f92500885f0 con 0x7f925810d390 2024-03-24T11:40:22.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.331+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6809/2539296009 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f924807cd70 con 0x7f9248079060 2024-03-24T11:40:22.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.331+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f9250088a20 con 0x7f925810d390 2024-03-24T11:40:22.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.332+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== osd.2 v1:172.21.15.114:6809/2539296009 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f924807cd70 con 0x7f9248079060 2024-03-24T11:40:22.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.351+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 --> v1:172.21.15.114:6809/2539296009 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f925806a200 con 0x7f9248079060 2024-03-24T11:40:22.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.351+0000 7f9255ffb700 1 -- 172.21.15.114:0/465421797 <== osd.2 v1:172.21.15.114:6809/2539296009 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2783450430) 0x7f925806a200 con 0x7f9248079060 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.351+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 >> v1:172.21.15.114:6809/2539296009 conn(0x7f9248079060 legacy=0x7f924807b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.352+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 >> v1:172.21.15.114:6800/1253397276 conn(0x7f924806e720 legacy=0x7f9248070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.352+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 >> v1:172.21.15.114:6789/0 conn(0x7f925810d390 legacy=0x7f92581a4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.352+0000 7f925d10c700 1 -- 172.21.15.114:0/465421797 reap_dead start 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.352+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 shutdown_connections 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.352+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 >> 172.21.15.114:0/465421797 conn(0x7f92580fc420 msgr2=0x7f92581108b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.352+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 shutdown_connections 2024-03-24T11:40:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:22.352+0000 7f925eb6f700 1 -- 172.21.15.114:0/465421797 wait complete. 2024-03-24T11:40:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:22 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:22.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:22 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:22 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:23.115 INFO:teuthology.orchestra.run.smithi114.stdout:167503724568 2024-03-24T11:40:23.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T11:40:23.717 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:23 smithi114 ceph-mon[30312]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:23.718 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:23 smithi114 ceph-mon[42466]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:23.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.727+0000 7f18096c7700 1 -- 172.21.15.114:0/2482180762 >> v1:172.21.15.114:6790/0 conn(0x7f180410e590 legacy=0x7f18041109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:23.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.727+0000 7f18096c7700 1 -- 172.21.15.114:0/2482180762 shutdown_connections 2024-03-24T11:40:23.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.727+0000 7f18096c7700 1 -- 172.21.15.114:0/2482180762 >> 172.21.15.114:0/2482180762 conn(0x7f18040fc460 msgr2=0x7f18040fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:23.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.727+0000 7f18096c7700 1 -- 172.21.15.114:0/2482180762 shutdown_connections 2024-03-24T11:40:23.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.727+0000 7f18096c7700 1 -- 172.21.15.114:0/2482180762 wait complete. 2024-03-24T11:40:23.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.728+0000 7f18096c7700 1 Processor -- start 2024-03-24T11:40:23.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.728+0000 7f18096c7700 1 -- start start 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f18096c7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18041a6990 con 0x7f1804102c70 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f18096c7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18041a7b70 con 0x7f180410e590 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f18096c7700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f18041a8d70 con 0x7f180410aa80 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f1802ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1804102c70 0x7f180410a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38642/0 (socket says 172.21.15.114:38642) 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f1802ffd700 1 -- 172.21.15.114:0/178763532 learned_addr learned my addr 172.21.15.114:0/178763532 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838773563 0 0) 0x7f18041a8d70 con 0x7f180410aa80 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ec0035b0 con 0x7f180410aa80 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003527711 0 0) 0x7f18041a6990 con 0x7f1804102c70 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18041a8d70 con 0x7f1804102c70 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3411317916 0 0) 0x7f17ec0035b0 con 0x7f180410aa80 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18041a6990 con 0x7f180410aa80 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f17f8001830 con 0x7f180410aa80 2024-03-24T11:40:23.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2264979405 0 0) 0x7f18041a8d70 con 0x7f1804102c70 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17ec0035b0 con 0x7f1804102c70 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493336089 0 0) 0x7f18041a7b70 con 0x7f180410e590 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18041a8d70 con 0x7f180410e590 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f17f4001970 con 0x7f1804102c70 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4037033057 0 0) 0x7f18041a6990 con 0x7f180410aa80 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 >> v1:172.21.15.119:6789/0 conn(0x7f180410e590 legacy=0x7f18041a5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 >> v1:172.21.15.114:6789/0 conn(0x7f1804102c70 legacy=0x7f180410a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18041a9f70 con 0x7f180410aa80 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18041a6b80 con 0x7f180410aa80 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.729+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18041a7000 con 0x7f180410aa80 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.730+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f17f8001d60 con 0x7f180410aa80 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.730+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f17f8004960 con 0x7f180410aa80 2024-03-24T11:40:23.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.730+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.114:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f17f0000f80 con 0x7f180410aa80 2024-03-24T11:40:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.731+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f17f8004ac0 con 0x7f180410aa80 2024-03-24T11:40:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.731+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f17f8087030 con 0x7f180410aa80 2024-03-24T11:40:23.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.731+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.119:6812/2022246932 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17ec07ca30 con 0x7f17ec078d20 2024-03-24T11:40:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.731+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f17f80873d0 con 0x7f180410aa80 2024-03-24T11:40:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.733+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== osd.7 v1:172.21.15.119:6812/2022246932 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17ec07ca30 con 0x7f17ec078d20 2024-03-24T11:40:23.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.750+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 --> v1:172.21.15.119:6812/2022246932 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f17f0002c40 con 0x7f17ec078d20 2024-03-24T11:40:23.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f17ebfff700 1 -- 172.21.15.114:0/178763532 <== osd.7 v1:172.21.15.119:6812/2022246932 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2936409065) 0x7f17f0002c40 con 0x7f17ec078d20 2024-03-24T11:40:23.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 >> v1:172.21.15.119:6812/2022246932 conn(0x7f17ec078d20 legacy=0x7f17ec07b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 >> v1:172.21.15.114:6800/1253397276 conn(0x7f17ec06e5c0 legacy=0x7f17ec070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 >> v1:172.21.15.114:6790/0 conn(0x7f180410aa80 legacy=0x7f18041a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18037fe700 1 -- 172.21.15.114:0/178763532 reap_dead start 2024-03-24T11:40:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 shutdown_connections 2024-03-24T11:40:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 >> 172.21.15.114:0/178763532 conn(0x7f18040fc460 msgr2=0x7f18040fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 shutdown_connections 2024-03-24T11:40:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:23.752+0000 7f18096c7700 1 -- 172.21.15.114:0/178763532 wait complete. 2024-03-24T11:40:23.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:23 smithi119 ceph-mon[33714]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:24.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.717+0000 7f47149ac700 1 -- 172.21.15.114:0/4136998313 >> v1:172.21.15.114:6790/0 conn(0x7f471010d380 legacy=0x7f471010f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.717+0000 7f47149ac700 1 -- 172.21.15.114:0/4136998313 shutdown_connections 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.717+0000 7f47149ac700 1 -- 172.21.15.114:0/4136998313 >> 172.21.15.114:0/4136998313 conn(0x7f47100fc410 msgr2=0x7f47100fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.717+0000 7f47149ac700 1 -- 172.21.15.114:0/4136998313 shutdown_connections 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.717+0000 7f47149ac700 1 -- 172.21.15.114:0/4136998313 wait complete. 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f47149ac700 1 Processor -- start 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f47149ac700 1 -- start start 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f47149ac700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47101a6950 con 0x7f471010d380 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f47149ac700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47101a7b30 con 0x7f47101097b0 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f47149ac700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47101a8d30 con 0x7f4710106900 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f470e59c700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4710106900 0x7f471010cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:44646/0 (socket says 172.21.15.114:44646) 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f470e59c700 1 -- 172.21.15.114:0/4182993113 learned_addr learned my addr 172.21.15.114:0/4182993113 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973650686 0 0) 0x7f47101a6950 con 0x7f471010d380 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f80035b0 con 0x7f471010d380 2024-03-24T11:40:24.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128621237 0 0) 0x7f47101a8d30 con 0x7f4710106900 2024-03-24T11:40:24.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47101a6950 con 0x7f4710106900 2024-03-24T11:40:24.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917721794 0 0) 0x7f47101a7b30 con 0x7f47101097b0 2024-03-24T11:40:24.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.718+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47101a8d30 con 0x7f47101097b0 2024-03-24T11:40:24.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 697680205 0 0) 0x7f46f80035b0 con 0x7f471010d380 2024-03-24T11:40:24.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47101a7b30 con 0x7f471010d380 2024-03-24T11:40:24.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2452083573 0 0) 0x7f47101a6950 con 0x7f4710106900 2024-03-24T11:40:24.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f80035b0 con 0x7f4710106900 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4708001cd0 con 0x7f471010d380 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4700001830 con 0x7f4710106900 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3072123705 0 0) 0x7f47101a8d30 con 0x7f47101097b0 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47101a6950 con 0x7f47101097b0 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f47040017d0 con 0x7f47101097b0 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301190521 0 0) 0x7f47101a7b30 con 0x7f471010d380 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 >> v1:172.21.15.114:6790/0 conn(0x7f4710106900 legacy=0x7f471010cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 >> v1:172.21.15.119:6789/0 conn(0x7f47101097b0 legacy=0x7f47101a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47101a9f30 con 0x7f471010d380 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47101a6b40 con 0x7f471010d380 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47101a6fc0 con 0x7f471010d380 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f47080014a0 con 0x7f471010d380 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.719+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4708004fc0 con 0x7f471010d380 2024-03-24T11:40:24.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.720+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.114:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46f0000f80 con 0x7f471010d380 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.720+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f4708003a00 con 0x7f471010d380 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.721+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f47080876c0 con 0x7f471010d380 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.721+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.119:6800/1375520258 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46f807ca90 con 0x7f46f8078d80 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.721+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f4708087ac0 con 0x7f471010d380 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.722+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== osd.4 v1:172.21.15.119:6800/1375520258 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f46f807ca90 con 0x7f46f8078d80 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.739+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 --> v1:172.21.15.119:6800/1375520258 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46f0002c40 con 0x7f46f8078d80 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f46ff7fe700 1 -- 172.21.15.114:0/4182993113 <== osd.4 v1:172.21.15.119:6800/1375520258 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1248179651) 0x7f46f0002c40 con 0x7f46f8078d80 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 >> v1:172.21.15.119:6800/1375520258 conn(0x7f46f8078d80 legacy=0x7f46f807b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 >> v1:172.21.15.114:6800/1253397276 conn(0x7f46f806e620 legacy=0x7f46f8070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:24.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 >> v1:172.21.15.114:6789/0 conn(0x7f471010d380 legacy=0x7f47101a5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:24.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f470ed9d700 1 -- 172.21.15.114:0/4182993113 reap_dead start 2024-03-24T11:40:24.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 shutdown_connections 2024-03-24T11:40:24.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 >> 172.21.15.114:0/4182993113 conn(0x7f47100fc410 msgr2=0x7f4710069b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:24.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 shutdown_connections 2024-03-24T11:40:24.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:20.740+0000 7f47149ac700 1 -- 172.21.15.114:0/4182993113 wait complete. 2024-03-24T11:40:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:40:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:25.028 INFO:teuthology.orchestra.run.smithi114.stdout:77309411370 2024-03-24T11:40:25.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:40:25.320 INFO:teuthology.orchestra.run.smithi114.stdout:133143986207 2024-03-24T11:40:25.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T11:40:25.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:25 smithi114 ceph-mon[30312]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:40:25.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:25 smithi114 ceph-mon[42466]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:40:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.591+0000 7f252b9e7700 1 -- 172.21.15.114:0/2981774672 >> v1:172.21.15.114:6789/0 conn(0x7f2524107b40 legacy=0x7f2524107f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.591+0000 7f252b9e7700 1 -- 172.21.15.114:0/2981774672 shutdown_connections 2024-03-24T11:40:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.591+0000 7f252b9e7700 1 -- 172.21.15.114:0/2981774672 >> 172.21.15.114:0/2981774672 conn(0x7f2524075c00 msgr2=0x7f2524076010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f252b9e7700 1 -- 172.21.15.114:0/2981774672 shutdown_connections 2024-03-24T11:40:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f252b9e7700 1 -- 172.21.15.114:0/2981774672 wait complete. 2024-03-24T11:40:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f252b9e7700 1 Processor -- start 2024-03-24T11:40:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f252b9e7700 1 -- start start 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f252b9e7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25241a8b30 con 0x7f252410e4b0 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f252b9e7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25241a9d10 con 0x7f2524107b40 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f252b9e7700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25241aaf10 con 0x7f252410a8b0 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f2528f82700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f252410a8b0 0x7f25241a3c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:44696/0 (socket says 172.21.15.114:44696) 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f2529f84700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f252410e4b0 0x7f25241a7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38670/0 (socket says 172.21.15.114:38670) 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.592+0000 7f2528f82700 1 -- 172.21.15.114:0/3087847373 learned_addr learned my addr 172.21.15.114:0/3087847373 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270524236 0 0) 0x7f25241aaf10 con 0x7f252410a8b0 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25100031f0 con 0x7f252410a8b0 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895148286 0 0) 0x7f25241a8b30 con 0x7f252410e4b0 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25241aaf10 con 0x7f252410e4b0 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154071671 0 0) 0x7f25241a9d10 con 0x7f2524107b40 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25241a8b30 con 0x7f2524107b40 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1280065946 0 0) 0x7f25100031f0 con 0x7f252410a8b0 2024-03-24T11:40:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25241a9d10 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f25140018f0 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1309886508 0 0) 0x7f25241aaf10 con 0x7f252410e4b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25100031f0 con 0x7f252410e4b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f251c001bd0 con 0x7f252410e4b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 308259805 0 0) 0x7f25241a8b30 con 0x7f2524107b40 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25241aaf10 con 0x7f2524107b40 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2520004480 con 0x7f2524107b40 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1087532377 0 0) 0x7f25241a9d10 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 >> v1:172.21.15.119:6789/0 conn(0x7f2524107b40 legacy=0x7f2524106bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 >> v1:172.21.15.114:6789/0 conn(0x7f252410e4b0 legacy=0x7f25241a7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25241ac110 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25241a8d20 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25241a91a0 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2514003b30 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.593+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2514005c90 con 0x7f252410a8b0 2024-03-24T11:40:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.594+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2508000f80 con 0x7f252410a8b0 2024-03-24T11:40:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.595+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f25140036a0 con 0x7f252410a8b0 2024-03-24T11:40:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.595+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f25140883c0 con 0x7f252410a8b0 2024-03-24T11:40:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.595+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6805/1920152174 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f251007ca50 con 0x7f2510078d40 2024-03-24T11:40:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.595+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f25140887f0 con 0x7f252410a8b0 2024-03-24T11:40:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.596+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== osd.1 v1:172.21.15.114:6805/1920152174 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f251007ca50 con 0x7f2510078d40 2024-03-24T11:40:25.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.612+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 --> v1:172.21.15.114:6805/1920152174 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2508002c40 con 0x7f2510078d40 2024-03-24T11:40:25.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.612+0000 7f251a7fc700 1 -- 172.21.15.114:0/3087847373 <== osd.1 v1:172.21.15.114:6805/1920152174 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3707702002) 0x7f2508002c40 con 0x7f2510078d40 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 >> v1:172.21.15.114:6805/1920152174 conn(0x7f2510078d40 legacy=0x7f251007b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 >> v1:172.21.15.114:6800/1253397276 conn(0x7f251006e660 legacy=0x7f2510070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 >> v1:172.21.15.114:6790/0 conn(0x7f252410a8b0 legacy=0x7f25241a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f2529f84700 1 -- 172.21.15.114:0/3087847373 reap_dead start 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 shutdown_connections 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 >> 172.21.15.114:0/3087847373 conn(0x7f2524075c00 msgr2=0x7f2524109070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 shutdown_connections 2024-03-24T11:40:25.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:25.613+0000 7f252b9e7700 1 -- 172.21.15.114:0/3087847373 wait complete. 2024-03-24T11:40:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:25 smithi119 ceph-mon[33714]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:40:26.420 INFO:teuthology.orchestra.run.smithi114.stdout:236223201296 2024-03-24T11:40:26.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T11:40:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:26 smithi114 ceph-mon[30312]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:26.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:26 smithi114 ceph-mon[42466]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:26 smithi119 ceph-mon[33714]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:27.780 INFO:teuthology.orchestra.run.smithi114.stdout:34359738420 2024-03-24T11:40:27.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:40:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.976+0000 7f2570501700 1 -- 172.21.15.114:0/505945601 >> v1:172.21.15.114:6790/0 conn(0x7f25681050b0 legacy=0x7f2568105490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.976+0000 7f2570501700 1 -- 172.21.15.114:0/505945601 shutdown_connections 2024-03-24T11:40:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.976+0000 7f2570501700 1 -- 172.21.15.114:0/505945601 >> 172.21.15.114:0/505945601 conn(0x7f2568076cf0 msgr2=0x7f2568077100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.976+0000 7f2570501700 1 -- 172.21.15.114:0/505945601 shutdown_connections 2024-03-24T11:40:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.976+0000 7f2570501700 1 -- 172.21.15.114:0/505945601 wait complete. 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.977+0000 7f2570501700 1 Processor -- start 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.977+0000 7f2570501700 1 -- start start 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.977+0000 7f2570501700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25681b57a0 con 0x7f2568108f30 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.977+0000 7f2570501700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25681b69a0 con 0x7f25681050b0 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.977+0000 7f2570501700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25681b7ba0 con 0x7f256810cb30 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.977+0000 7f256da9c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2568108f30 0x7f2568109a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38686/0 (socket says 172.21.15.114:38686) 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.977+0000 7f256da9c700 1 -- 172.21.15.114:0/2889555400 learned_addr learned my addr 172.21.15.114:0/2889555400 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366572613 0 0) 0x7f25681b57a0 con 0x7f2568108f30 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2554005af0 con 0x7f2568108f30 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822465585 0 0) 0x7f25681b7ba0 con 0x7f256810cb30 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25681b57a0 con 0x7f256810cb30 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040368536 0 0) 0x7f25681b69a0 con 0x7f25681050b0 2024-03-24T11:40:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25681b7ba0 con 0x7f25681050b0 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566986794 0 0) 0x7f2554005af0 con 0x7f2568108f30 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25681b69a0 con 0x7f2568108f30 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2558001970 con 0x7f2568108f30 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2062707131 0 0) 0x7f25681b57a0 con 0x7f256810cb30 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2554005af0 con 0x7f256810cb30 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2560001470 con 0x7f256810cb30 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3478875710 0 0) 0x7f25681b69a0 con 0x7f2568108f30 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 >> v1:172.21.15.114:6790/0 conn(0x7f256810cb30 legacy=0x7f25680736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 >> v1:172.21.15.119:6789/0 conn(0x7f25681050b0 legacy=0x7f2568109310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25681b8da0 con 0x7f2568108f30 2024-03-24T11:40:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25681b7d90 con 0x7f2568108f30 2024-03-24T11:40:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.978+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25681b8290 con 0x7f2568108f30 2024-03-24T11:40:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.979+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2558003b10 con 0x7f2568108f30 2024-03-24T11:40:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.979+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2558005c60 con 0x7f2568108f30 2024-03-24T11:40:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.979+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.114:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f254c000f80 con 0x7f2568108f30 2024-03-24T11:40:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.980+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f255801c270 con 0x7f2568108f30 2024-03-24T11:40:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.981+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f2558088140 con 0x7f2568108f30 2024-03-24T11:40:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.981+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.119:6808/1563073715 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f255407c960 con 0x7f2554078c50 2024-03-24T11:40:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.981+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f2558005c60 con 0x7f2568108f30 2024-03-24T11:40:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:27.983+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== osd.6 v1:172.21.15.119:6808/1563073715 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f255407c960 con 0x7f2554078c50 2024-03-24T11:40:28.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.001+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 --> v1:172.21.15.119:6808/1563073715 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f254c002c40 con 0x7f2554078c50 2024-03-24T11:40:28.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.001+0000 7f255f7fe700 1 -- 172.21.15.114:0/2889555400 <== osd.6 v1:172.21.15.119:6808/1563073715 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 4240359720) 0x7f254c002c40 con 0x7f2554078c50 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 >> v1:172.21.15.119:6808/1563073715 conn(0x7f2554078c50 legacy=0x7f255407b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 >> v1:172.21.15.114:6800/1253397276 conn(0x7f255406e570 legacy=0x7f2554070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 >> v1:172.21.15.114:6789/0 conn(0x7f2568108f30 legacy=0x7f2568109a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f256ea9e700 1 -- 172.21.15.114:0/2889555400 reap_dead start 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 shutdown_connections 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 >> 172.21.15.114:0/2889555400 conn(0x7f2568076cf0 msgr2=0x7f2568108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 shutdown_connections 2024-03-24T11:40:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.002+0000 7f2570501700 1 -- 172.21.15.114:0/2889555400 wait complete. 2024-03-24T11:40:28.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.236+0000 7f32cd9ca700 1 -- 172.21.15.114:0/3244156665 >> v1:172.21.15.114:6789/0 conn(0x7f32c8107a30 legacy=0x7f32c8107e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.236+0000 7f32cd9ca700 1 -- 172.21.15.114:0/3244156665 shutdown_connections 2024-03-24T11:40:28.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.236+0000 7f32cd9ca700 1 -- 172.21.15.114:0/3244156665 >> 172.21.15.114:0/3244156665 conn(0x7f32c80fd560 msgr2=0x7f32c80ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:28.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.237+0000 7f32cd9ca700 1 -- 172.21.15.114:0/3244156665 shutdown_connections 2024-03-24T11:40:28.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.237+0000 7f32cd9ca700 1 -- 172.21.15.114:0/3244156665 wait complete. 2024-03-24T11:40:28.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.238+0000 7f32cd9ca700 1 Processor -- start 2024-03-24T11:40:28.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.238+0000 7f32cd9ca700 1 -- start start 2024-03-24T11:40:28.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32c6ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f32c8107a30 0x7f32c810da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38698/0 (socket says 172.21.15.114:38698) 2024-03-24T11:40:28.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32c6ffd700 1 -- 172.21.15.114:0/1711181402 learned_addr learned my addr 172.21.15.114:0/1711181402 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32c81a67b0 con 0x7f32c8107a30 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32c81a6940 con 0x7f32c810e4b0 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32c81a6b10 con 0x7f32c810a8e0 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258490569 0 0) 0x7f32c81a67b0 con 0x7f32c8107a30 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b00035b0 con 0x7f32c8107a30 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827003934 0 0) 0x7f32c81a6b10 con 0x7f32c810a8e0 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c81a67b0 con 0x7f32c810a8e0 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527707768 0 0) 0x7f32c81a6940 con 0x7f32c810e4b0 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c81a6b10 con 0x7f32c810e4b0 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3213127373 0 0) 0x7f32b00035b0 con 0x7f32c8107a30 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32c81a6940 con 0x7f32c8107a30 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.239+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f32b8004480 con 0x7f32c8107a30 2024-03-24T11:40:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1159657322 0 0) 0x7f32c81a6940 con 0x7f32c8107a30 2024-03-24T11:40:28.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 >> v1:172.21.15.114:6790/0 conn(0x7f32c810a8e0 legacy=0x7f32c81a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 >> v1:172.21.15.119:6789/0 conn(0x7f32c810e4b0 legacy=0x7f32c81a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c81a6d00 con 0x7f32c8107a30 2024-03-24T11:40:28.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c81a6fb0 con 0x7f32c8107a30 2024-03-24T11:40:28.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c81a7430 con 0x7f32c8107a30 2024-03-24T11:40:28.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f32b8001a10 con 0x7f32c8107a30 2024-03-24T11:40:28.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.240+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f32b8004f00 con 0x7f32c8107a30 2024-03-24T11:40:28.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.241+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32c8112440 con 0x7f32c8107a30 2024-03-24T11:40:28.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.242+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f32b8005060 con 0x7f32c8107a30 2024-03-24T11:40:28.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.243+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f32b8088670 con 0x7f32c8107a30 2024-03-24T11:40:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.243+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6813/3141649135 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b0080920 con 0x7f32b007ccb0 2024-03-24T11:40:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.243+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (unknown 571355762 0 0) 0x7f32b80889c0 con 0x7f32c8107a30 2024-03-24T11:40:28.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.245+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== osd.3 v1:172.21.15.114:6813/3141649135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f32b0080920 con 0x7f32b007ccb0 2024-03-24T11:40:28.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.264+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 --> v1:172.21.15.114:6813/3141649135 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32c8103170 con 0x7f32b007ccb0 2024-03-24T11:40:28.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.264+0000 7f32cc9c8700 1 -- 172.21.15.114:0/1711181402 <== osd.3 v1:172.21.15.114:6813/3141649135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2404460167) 0x7f32c8103170 con 0x7f32b007ccb0 2024-03-24T11:40:28.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 >> v1:172.21.15.114:6813/3141649135 conn(0x7f32b007ccb0 legacy=0x7f32b007f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 >> v1:172.21.15.114:6800/1253397276 conn(0x7f32b00724b0 legacy=0x7f32b0074970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 >> v1:172.21.15.114:6789/0 conn(0x7f32c8107a30 legacy=0x7f32c810da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:28.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32c77fe700 1 -- 172.21.15.114:0/1711181402 reap_dead start 2024-03-24T11:40:28.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 shutdown_connections 2024-03-24T11:40:28.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 >> 172.21.15.114:0/1711181402 conn(0x7f32c80fd560 msgr2=0x7f32c8111120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:28.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 shutdown_connections 2024-03-24T11:40:28.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:28.265+0000 7f32cd9ca700 1 -- 172.21.15.114:0/1711181402 wait complete. 2024-03-24T11:40:29.054 INFO:teuthology.orchestra.run.smithi114.stdout:201863462935 2024-03-24T11:40:29.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T11:40:29.444 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 podman[78058]: 2024-03-24 11:40:29.271432632 +0000 UTC m=+0.828248355 container create b46fb4351c0a063f27bb81d8042ec68eb2a45b0c797539f3e21278010a46facf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-mon[30312]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-mon[42466]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:29.739 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 podman[78058]: 2024-03-24 11:40:29.557701419 +0000 UTC m=+1.114517149 container init b46fb4351c0a063f27bb81d8042ec68eb2a45b0c797539f3e21278010a46facf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 podman[78058]: 2024-03-24 11:40:29.563479443 +0000 UTC m=+1.120295174 container start b46fb4351c0a063f27bb81d8042ec68eb2a45b0c797539f3e21278010a46facf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.575Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.575Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.576Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.114 port=9094 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.577Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.600Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.600Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.603Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:29.603Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 bash[78058]: b46fb4351c0a063f27bb81d8042ec68eb2a45b0c797539f3e21278010a46facf 2024-03-24T11:40:29.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:29 smithi114 systemd[1]: Started Ceph alertmanager.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:40:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:29 smithi119 ceph-mon[33714]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:30.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:29] ENGINE Bus STOPPING 2024-03-24T11:40:30.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:40:30.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:29] ENGINE Bus STOPPED 2024-03-24T11:40:30.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:29] ENGINE Bus STARTING 2024-03-24T11:40:30.240 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:29] ENGINE Serving on http://:::9283 2024-03-24T11:40:30.240 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:40:29] ENGINE Bus STARTED 2024-03-24T11:40:30.302 INFO:teuthology.orchestra.run.smithi114.stdout:55834574894 2024-03-24T11:40:30.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:40:30.864 INFO:teuthology.orchestra.run.smithi114.stdout:111669149734 2024-03-24T11:40:30.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T11:40:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:30.946 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:40:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: Deploying daemon grafana.a on smithi119 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: Deploying daemon grafana.a on smithi119 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:40:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:31.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:31.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: Deploying daemon grafana.a on smithi119 2024-03-24T11:40:31.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:31.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:31.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:31.595 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:31 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:31.577Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00027047s 2024-03-24T11:40:31.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:40:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:40:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:31 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:32.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:31 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:32.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:31 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:33.186 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:32 smithi114 ceph-mon[30312]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:32 smithi114 ceph-mon[42466]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:32 smithi119 ceph-mon[33714]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:34.029 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:34.956 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:40:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:35 smithi114 ceph-mon[30312]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:35.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:35 smithi114 ceph-mon[42466]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:35 smithi119 ceph-mon[33714]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:36.332 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.958+0000 7f466c4e8700 1 -- 172.21.15.114:0/1946819471 >> v1:172.21.15.114:6789/0 conn(0x7f4664074b00 legacy=0x7f4664113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- 172.21.15.114:0/1946819471 shutdown_connections 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- 172.21.15.114:0/1946819471 >> 172.21.15.114:0/1946819471 conn(0x7f46640fd550 msgr2=0x7f46640ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- 172.21.15.114:0/1946819471 shutdown_connections 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- 172.21.15.114:0/1946819471 wait complete. 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 Processor -- start 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- start start 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46641ace00 con 0x7f4664074b00 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46641adfe0 con 0x7f4664074010 2024-03-24T11:40:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466c4e8700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46641af1e0 con 0x7f4664114c20 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f4669a83700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4664074b00 0x7f46641a7e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48302/0 (socket says 172.21.15.114:48302) 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f466aa85700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4664114c20 0x7f46641ab560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34314/0 (socket says 172.21.15.114:34314) 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.959+0000 7f4669a83700 1 -- 172.21.15.114:0/2377022199 learned_addr learned my addr 172.21.15.114:0/2377022199 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971965522 0 0) 0x7f46641ace00 con 0x7f4664074b00 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46500049e0 con 0x7f4664074b00 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475272056 0 0) 0x7f46641af1e0 con 0x7f4664114c20 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46641ace00 con 0x7f4664114c20 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122535629 0 0) 0x7f46641adfe0 con 0x7f4664074010 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46641af1e0 con 0x7f4664074010 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635684388 0 0) 0x7f46641af1e0 con 0x7f4664074010 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46641adfe0 con 0x7f4664074010 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712620499 0 0) 0x7f46500049e0 con 0x7f4664074b00 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46641af1e0 con 0x7f4664074b00 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3853492865 0 0) 0x7f46641ace00 con 0x7f4664114c20 2024-03-24T11:40:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46500049e0 con 0x7f4664114c20 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4660001850 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4654001340 con 0x7f4664074b00 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f465c001bb0 con 0x7f4664114c20 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1098923188 0 0) 0x7f46641adfe0 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 >> v1:172.21.15.114:6790/0 conn(0x7f4664114c20 legacy=0x7f46641ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 >> v1:172.21.15.114:6789/0 conn(0x7f4664074b00 legacy=0x7f46641a7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46641b03e0 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46641af3d0 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46641af850 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f4660001d20 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.960+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4660004860 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.961+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4648005120 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.962+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f4660003680 con 0x7f4664074010 2024-03-24T11:40:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.962+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f4660086fe0 con 0x7f4664074010 2024-03-24T11:40:36.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.967+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f46600544e0 con 0x7f4664074010 2024-03-24T11:40:37.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.161+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f46480053d0 con 0x7f4664074010 2024-03-24T11:40:37.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.161+0000 7f465b7fe700 1 -- 172.21.15.114:0/2377022199 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 1292561331) 0x7f4660052850 con 0x7f4664074010 2024-03-24T11:40:37.162 INFO:teuthology.orchestra.run.smithi114.stdout:167503724574 2024-03-24T11:40:37.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.164+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 >> v1:172.21.15.114:6800/1253397276 conn(0x7f465006e6b0 legacy=0x7f4650070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:37.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.164+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 >> v1:172.21.15.119:6789/0 conn(0x7f4664074010 legacy=0x7f46641141b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:37.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.164+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 shutdown_connections 2024-03-24T11:40:37.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.164+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 >> 172.21.15.114:0/2377022199 conn(0x7f46640fd550 msgr2=0x7f46640ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:37.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.164+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 shutdown_connections 2024-03-24T11:40:37.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:37.164+0000 7f466c4e8700 1 -- 172.21.15.114:0/2377022199 wait complete. 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.341+0000 7f142ac14700 1 -- 172.21.15.114:0/549001562 >> v1:172.21.15.114:6790/0 conn(0x7f1424106980 legacy=0x7f1424106d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.341+0000 7f142ac14700 1 -- 172.21.15.114:0/549001562 shutdown_connections 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.341+0000 7f142ac14700 1 -- 172.21.15.114:0/549001562 >> 172.21.15.114:0/549001562 conn(0x7f1424076da0 msgr2=0x7f14240fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.341+0000 7f142ac14700 1 -- 172.21.15.114:0/549001562 shutdown_connections 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.341+0000 7f142ac14700 1 -- 172.21.15.114:0/549001562 wait complete. 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.342+0000 7f142ac14700 1 Processor -- start 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.342+0000 7f142ac14700 1 -- start start 2024-03-24T11:40:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f142ac14700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14241a5760 con 0x7f1424109830 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f142ac14700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14241a6940 con 0x7f1424106980 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f142ac14700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14241a7b40 con 0x7f142410d400 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f14289b0700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1424106980 0x7f142410c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.114:35876/0 (socket says 172.21.15.114:35876) 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f14289b0700 1 -- 172.21.15.114:0/4023891616 learned_addr learned my addr 172.21.15.114:0/4023891616 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102296054 0 0) 0x7f14241a6940 con 0x7f1424106980 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14140049e0 con 0x7f1424106980 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509560592 0 0) 0x7f14241a7b40 con 0x7f142410d400 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14241a6940 con 0x7f142410d400 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590596834 0 0) 0x7f14241a5760 con 0x7f1424109830 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14241a7b40 con 0x7f1424109830 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278675324 0 0) 0x7f14140049e0 con 0x7f1424106980 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.343+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14241a5760 con 0x7f1424106980 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f1418001390 con 0x7f1424106980 2024-03-24T11:40:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384869136 0 0) 0x7f14241a6940 con 0x7f142410d400 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14140049e0 con 0x7f142410d400 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976295183 0 0) 0x7f14241a7b40 con 0x7f1424109830 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14241a6940 con 0x7f1424109830 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f141c001590 con 0x7f142410d400 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f1410001770 con 0x7f1424109830 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3311968949 0 0) 0x7f14241a5760 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 >> v1:172.21.15.114:6790/0 conn(0x7f142410d400 legacy=0x7f14241a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 >> v1:172.21.15.114:6789/0 conn(0x7f1424109830 legacy=0x7f14241a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14241a8d40 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14241a7d30 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f1418001a50 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f1418004f70 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.344+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14241a81b0 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.345+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1424068560 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.346+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f14180050d0 con 0x7f1424106980 2024-03-24T11:40:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.346+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f14180886e0 con 0x7f1424106980 2024-03-24T11:40:37.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.355+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f1418055ca0 con 0x7f1424106980 2024-03-24T11:40:37.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.562+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f142410afe0 con 0x7f1424106980 2024-03-24T11:40:37.185 INFO:teuthology.orchestra.run.smithi114.stdout:133143986211 2024-03-24T11:40:37.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.562+0000 7f1421ffb700 1 -- 172.21.15.114:0/4023891616 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 2012961777) 0x7f1418001130 con 0x7f1424106980 2024-03-24T11:40:37.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.564+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 >> v1:172.21.15.114:6800/1253397276 conn(0x7f141406e8c0 legacy=0x7f1414070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:37.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.564+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 >> v1:172.21.15.119:6789/0 conn(0x7f1424106980 legacy=0x7f142410c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:37.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.564+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 shutdown_connections 2024-03-24T11:40:37.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.564+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 >> 172.21.15.114:0/4023891616 conn(0x7f1424076da0 msgr2=0x7f142410d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:37.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.564+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 shutdown_connections 2024-03-24T11:40:37.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:36.564+0000 7f142ac14700 1 -- 172.21.15.114:0/4023891616 wait complete. 2024-03-24T11:40:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:37 smithi114 ceph-mon[30312]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:40:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4023891616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:40:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2377022199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:40:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:37 smithi114 ceph-mon[42466]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:40:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4023891616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:40:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2377022199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:40:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:37 smithi119 ceph-mon[33714]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:40:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4023891616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:40:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2377022199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:40:37.839 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986207 got 133143986211 for osd.4 2024-03-24T11:40:37.839 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.377+0000 7f0631893700 1 -- 172.21.15.114:0/221073289 >> v1:172.21.15.114:6790/0 conn(0x7f062c105930 legacy=0x7f062c105d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.377+0000 7f0631893700 1 -- 172.21.15.114:0/221073289 shutdown_connections 2024-03-24T11:40:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.377+0000 7f0631893700 1 -- 172.21.15.114:0/221073289 >> 172.21.15.114:0/221073289 conn(0x7f062c076da0 msgr2=0x7f062c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.378+0000 7f0631893700 1 -- 172.21.15.114:0/221073289 shutdown_connections 2024-03-24T11:40:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.378+0000 7f0631893700 1 -- 172.21.15.114:0/221073289 wait complete. 2024-03-24T11:40:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.378+0000 7f0631893700 1 Processor -- start 2024-03-24T11:40:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.378+0000 7f0631893700 1 -- start start 2024-03-24T11:40:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0631893700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f062c1a67f0 con 0x7f062c1097b0 2024-03-24T11:40:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0631893700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f062c1a79d0 con 0x7f062c10d380 2024-03-24T11:40:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0631893700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f062c1a8bd0 con 0x7f062c105930 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f062affd700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f062c105930 0x7f062c10ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34324/0 (socket says 172.21.15.114:34324) 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f062affd700 1 -- 172.21.15.114:0/2808916421 learned_addr learned my addr 172.21.15.114:0/2808916421 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954046998 0 0) 0x7f062c1a8bd0 con 0x7f062c105930 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06140035b0 con 0x7f062c105930 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528447851 0 0) 0x7f062c1a79d0 con 0x7f062c10d380 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f062c1a8bd0 con 0x7f062c10d380 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001118321 0 0) 0x7f062c1a67f0 con 0x7f062c1097b0 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f062c1a79d0 con 0x7f062c1097b0 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 677071633 0 0) 0x7f06140035b0 con 0x7f062c105930 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f062c1a67f0 con 0x7f062c105930 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f061c004580 con 0x7f062c105930 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 392241477 0 0) 0x7f062c1a79d0 con 0x7f062c1097b0 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06140035b0 con 0x7f062c1097b0 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0620001870 con 0x7f062c1097b0 2024-03-24T11:40:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2952768205 0 0) 0x7f062c1a8bd0 con 0x7f062c10d380 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f062c1a79d0 con 0x7f062c10d380 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f06240014b0 con 0x7f062c10d380 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1471108437 0 0) 0x7f062c1a67f0 con 0x7f062c105930 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 >> v1:172.21.15.119:6789/0 conn(0x7f062c10d380 legacy=0x7f062c1a4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 >> v1:172.21.15.114:6789/0 conn(0x7f062c1097b0 legacy=0x7f062c1a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c1a9dd0 con 0x7f062c105930 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.379+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062c1a69e0 con 0x7f062c105930 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.380+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f062c1a6ee0 con 0x7f062c105930 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.380+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f061c001ab0 con 0x7f062c105930 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.380+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f061c004fd0 con 0x7f062c105930 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.380+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0618005120 con 0x7f062c105930 2024-03-24T11:40:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.381+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f061c005130 con 0x7f062c105930 2024-03-24T11:40:38.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.381+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f061c088640 con 0x7f062c105930 2024-03-24T11:40:38.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.386+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f061c055b20 con 0x7f062c105930 2024-03-24T11:40:38.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.586+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f06180053d0 con 0x7f062c105930 2024-03-24T11:40:38.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.586+0000 7f0630891700 1 -- 172.21.15.114:0/2808916421 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 3862917126) 0x7f061c053e90 con 0x7f062c105930 2024-03-24T11:40:38.587 INFO:teuthology.orchestra.run.smithi114.stdout:201863462937 2024-03-24T11:40:38.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.588+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 >> v1:172.21.15.114:6800/1253397276 conn(0x7f061406e6c0 legacy=0x7f0614070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:38.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.588+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 >> v1:172.21.15.114:6790/0 conn(0x7f062c105930 legacy=0x7f062c10ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:38.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.588+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 shutdown_connections 2024-03-24T11:40:38.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.588+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 >> 172.21.15.114:0/2808916421 conn(0x7f062c076da0 msgr2=0x7f062c1108a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:38.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.589+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 shutdown_connections 2024-03-24T11:40:38.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:38.589+0000 7f0631893700 1 -- 172.21.15.114:0/2808916421 wait complete. 2024-03-24T11:40:39.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:39 smithi114 ceph-mon[30312]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:39.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2808916421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:40:39.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:39 smithi114 ceph-mon[42466]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:39.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2808916421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:40:39.740 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:40:39 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:40:39.579Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002262672s 2024-03-24T11:40:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:39 smithi119 ceph-mon[33714]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2808916421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:40:40.746 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:41.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462935 got 201863462937 for osd.6 2024-03-24T11:40:41.136 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:41.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:41 smithi119 ceph-mon[33714]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:41.470 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:40:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:40:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:41 smithi114 ceph-mon[30312]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:41.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:41 smithi114 ceph-mon[42466]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:42.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:42.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:42.886 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724568 got 167503724574 for osd.5 2024-03-24T11:40:42.886 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.069+0000 7f645fee2700 1 -- 172.21.15.114:0/3677338302 >> v1:172.21.15.114:6790/0 conn(0x7f645810e490 legacy=0x7f6458110950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.069+0000 7f645fee2700 1 -- 172.21.15.114:0/3677338302 shutdown_connections 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.069+0000 7f645fee2700 1 -- 172.21.15.114:0/3677338302 >> 172.21.15.114:0/3677338302 conn(0x7f64580fd520 msgr2=0x7f64580ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.070+0000 7f645fee2700 1 -- 172.21.15.114:0/3677338302 shutdown_connections 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.070+0000 7f645fee2700 1 -- 172.21.15.114:0/3677338302 wait complete. 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645fee2700 1 Processor -- start 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645fee2700 1 -- start start 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645fee2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64581a8b10 con 0x7f645810e490 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645fee2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64581a9cb0 con 0x7f6458107a10 2024-03-24T11:40:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645fee2700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64581aaeb0 con 0x7f645810a8c0 2024-03-24T11:40:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645d47d700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f645810a8c0 0x7f64581a3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33316/0 (socket says 172.21.15.114:33316) 2024-03-24T11:40:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645e47f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f645810e490 0x7f64581a7360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43440/0 (socket says 172.21.15.114:43440) 2024-03-24T11:40:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f645d47d700 1 -- 172.21.15.114:0/2654355151 learned_addr learned my addr 172.21.15.114:0/2654355151 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808746331 0 0) 0x7f64581aaeb0 con 0x7f645810a8c0 2024-03-24T11:40:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64440031f0 con 0x7f645810a8c0 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112114427 0 0) 0x7f64581a9cb0 con 0x7f6458107a10 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.071+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64581aaeb0 con 0x7f6458107a10 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1083787434 0 0) 0x7f64581aaeb0 con 0x7f6458107a10 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64581a9cb0 con 0x7f6458107a10 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641899848 0 0) 0x7f64581a8b10 con 0x7f645810e490 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64581aaeb0 con 0x7f645810e490 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f6454001050 con 0x7f6458107a10 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 720777127 0 0) 0x7f64440031f0 con 0x7f645810a8c0 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64581a8b10 con 0x7f645810a8c0 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f6448001870 con 0x7f645810a8c0 2024-03-24T11:40:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3371633537 0 0) 0x7f64581a9cb0 con 0x7f6458107a10 2024-03-24T11:40:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 >> v1:172.21.15.114:6790/0 conn(0x7f645810a8c0 legacy=0x7f64581a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 >> v1:172.21.15.114:6789/0 conn(0x7f645810e490 legacy=0x7f64581a7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64581ac0b0 con 0x7f6458107a10 2024-03-24T11:40:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64581a9ea0 con 0x7f6458107a10 2024-03-24T11:40:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f6454001d40 con 0x7f6458107a10 2024-03-24T11:40:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f6454004910 con 0x7f6458107a10 2024-03-24T11:40:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.072+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64581aa320 con 0x7f6458107a10 2024-03-24T11:40:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.073+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6458103150 con 0x7f6458107a10 2024-03-24T11:40:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.074+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f6454003680 con 0x7f6458107a10 2024-03-24T11:40:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.074+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f6454087110 con 0x7f6458107a10 2024-03-24T11:40:43.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.079+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f64540545f0 con 0x7f6458107a10 2024-03-24T11:40:43.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.868+0000 7fdf12d0a700 1 -- 172.21.15.114:0/700992736 >> v1:172.21.15.114:6789/0 conn(0x7fdf0c10d390 legacy=0x7fdf0c10f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.868+0000 7fdf12d0a700 1 -- 172.21.15.114:0/700992736 shutdown_connections 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.868+0000 7fdf12d0a700 1 -- 172.21.15.114:0/700992736 >> 172.21.15.114:0/700992736 conn(0x7fdf0c0fc420 msgr2=0x7fdf0c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.868+0000 7fdf12d0a700 1 -- 172.21.15.114:0/700992736 shutdown_connections 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.868+0000 7fdf12d0a700 1 -- 172.21.15.114:0/700992736 wait complete. 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf12d0a700 1 Processor -- start 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf12d0a700 1 -- start start 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf12d0a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf0c1a8b50 con 0x7fdf0c10d390 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf12d0a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf0c1a9d30 con 0x7fdf0c1097c0 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf12d0a700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf0c1aaf30 con 0x7fdf0c106910 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf10aa6700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdf0c106910 0x7fdf0c10c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33300/0 (socket says 172.21.15.114:33300) 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf10aa6700 1 -- 172.21.15.114:0/3356910711 learned_addr learned my addr 172.21.15.114:0/3356910711 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635342519 0 0) 0x7fdf0c1aaf30 con 0x7fdf0c106910 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc0035b0 con 0x7fdf0c106910 2024-03-24T11:40:43.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523578560 0 0) 0x7fdf0c1a8b50 con 0x7fdf0c10d390 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0c1aaf30 con 0x7fdf0c10d390 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263695952 0 0) 0x7fdf0c1a9d30 con 0x7fdf0c1097c0 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0c1a8b50 con 0x7fdf0c1097c0 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1442371747 0 0) 0x7fdf0c1a8b50 con 0x7fdf0c1097c0 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf0c1a9d30 con 0x7fdf0c1097c0 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.869+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdef80017b0 con 0x7fdf0c1097c0 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2947102288 0 0) 0x7fdf0c1aaf30 con 0x7fdf0c10d390 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stdout:34359738425 2024-03-24T11:40:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf0c1a8b50 con 0x7fdf0c10d390 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669577779 0 0) 0x7fdefc0035b0 con 0x7fdf0c106910 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf0c1aaf30 con 0x7fdf0c106910 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdf040017b0 con 0x7fdf0c10d390 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdf00001a90 con 0x7fdf0c106910 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3449387692 0 0) 0x7fdf0c1a9d30 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 >> v1:172.21.15.114:6790/0 conn(0x7fdf0c106910 legacy=0x7fdf0c10c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 >> v1:172.21.15.114:6789/0 conn(0x7fdf0c10d390 legacy=0x7fdf0c1a7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c1ac130 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c1ab120 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fdef8001cc0 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fdef80048f0 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.870+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c1ab4e0 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.871+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c102050 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.871+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7fdef801afb0 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.872+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7fdef8087230 con 0x7fdf0c1097c0 2024-03-24T11:40:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:42.877+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fdef8054710 con 0x7fdf0c1097c0 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.088+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdf0c068560 con 0x7fdf0c1097c0 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.089+0000 7fdf09ffb700 1 -- 172.21.15.114:0/3356910711 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 1172101649) 0x7fdef8052a80 con 0x7fdf0c1097c0 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.091+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 >> v1:172.21.15.114:6800/1253397276 conn(0x7fdefc06e870 legacy=0x7fdefc070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.091+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 >> v1:172.21.15.119:6789/0 conn(0x7fdf0c1097c0 legacy=0x7fdf0c1a3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.091+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 shutdown_connections 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.091+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 >> 172.21.15.114:0/3356910711 conn(0x7fdf0c0fc420 msgr2=0x7fdf0c100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.091+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 shutdown_connections 2024-03-24T11:40:43.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.091+0000 7fdf12d0a700 1 -- 172.21.15.114:0/3356910711 wait complete. 2024-03-24T11:40:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.282+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6458068560 con 0x7f6458107a10 2024-03-24T11:40:43.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.282+0000 7f644effd700 1 -- 172.21.15.114:0/2654355151 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3109443431) 0x7f6454052960 con 0x7f6458107a10 2024-03-24T11:40:43.283 INFO:teuthology.orchestra.run.smithi114.stdout:77309411374 2024-03-24T11:40:43.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.284+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 >> v1:172.21.15.114:6800/1253397276 conn(0x7f644406e610 legacy=0x7f6444070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.284+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 >> v1:172.21.15.119:6789/0 conn(0x7f6458107a10 legacy=0x7f6458075870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.284+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 shutdown_connections 2024-03-24T11:40:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.284+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 >> 172.21.15.114:0/2654355151 conn(0x7f64580fd520 msgr2=0x7f6458101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.285+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 shutdown_connections 2024-03-24T11:40:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:43.285+0000 7f645fee2700 1 -- 172.21.15.114:0/2654355151 wait complete. 2024-03-24T11:40:43.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:43 smithi114 ceph-mon[30312]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:43.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3356910711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:40:43.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2654355151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:40:43.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:43 smithi114 ceph-mon[42466]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:43.695 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3356910711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:40:43.695 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2654355151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:40:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:43 smithi119 ceph-mon[33714]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3356910711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:40:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2654355151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:40:44.173 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411370 got 77309411374 for osd.2 2024-03-24T11:40:44.173 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:44.465 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 systemd[1]: Starting Ceph grafana.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:40:44.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738425 for osd.0 2024-03-24T11:40:44.574 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:44 smithi114 ceph-mon[30312]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:44.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:44 smithi114 ceph-mon[42466]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:44.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:40:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:44.799 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-mon[33714]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:44.799 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 podman[66415]: 2024-03-24 11:40:44.430860295 +0000 UTC m=+0.017150438 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-24T11:40:44.799 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 podman[66415]: 2024-03-24 11:40:44.555554578 +0000 UTC m=+0.141844767 container create c52fd27442fb885917fe2dd72f1fe9eb2623b7c68f63324e2eab8281a1d2746d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.expose-services=, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, description=Ceph Grafana Container, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, version=8.5) 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 podman[66415]: 2024-03-24 11:40:44.799301953 +0000 UTC m=+0.385592107 container init c52fd27442fb885917fe2dd72f1fe9eb2623b7c68f63324e2eab8281a1d2746d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, description=Ceph Grafana Container, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 podman[66415]: 2024-03-24 11:40:44.804412917 +0000 UTC m=+0.390703060 container start c52fd27442fb885917fe2dd72f1fe9eb2623b7c68f63324e2eab8281a1d2746d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, version=8.5, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, release=239.1651231664, distribution-scope=public) 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 bash[66415]: c52fd27442fb885917fe2dd72f1fe9eb2623b7c68f63324e2eab8281a1d2746d 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 systemd[1]: Started Ceph grafana.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958345191Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958551148Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958572955Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-24T11:40:45.050 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958580888Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958587423Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958593685Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958600631Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958606683Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958612815Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958618794Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958624547Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958630673Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958636604Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=settings t=2024-03-24T11:40:44.958642443Z level=info msg="App mode production" 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=sqlstore t=2024-03-24T11:40:44.958691224Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=sqlstore t=2024-03-24T11:40:44.958708274Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.95950757Z level=info msg="Starting DB migrations" 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.959582563Z level=info msg="Executing migration" id="create migration_log table" 2024-03-24T11:40:45.051 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.960080171Z level=info msg="Executing migration" id="create user table" 2024-03-24T11:40:45.052 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.960508163Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-24T11:40:45.052 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.9612545Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-24T11:40:45.052 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.962044284Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-24T11:40:45.052 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.962880108Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-24T11:40:45.052 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.963667515Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-24T11:40:45.052 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.964488527Z level=info msg="Executing migration" id="create user table v2" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.965233267Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.965627905Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.966489241Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.967282954Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.968182389Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.968842218Z level=info msg="Executing migration" id="Update user table charset" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.969657162Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.970101784Z level=info msg="Executing migration" id="Add missing user data" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.970542635Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.971267691Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.97167275Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.972152463Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.973269347Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.973689498Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.974688104Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T11:40:45.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.975335413Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.976173961Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.976577457Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.977314768Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.977693351Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.978197002Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.978597472Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.978938241Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.979553219Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.979932769Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.980946827Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.98163406Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.981987895Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.982481699Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.984639898Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-24T11:40:45.054 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.987401249Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.990148602Z level=info msg="Executing migration" id="create star table" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.993130108Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.995755075Z level=info msg="Executing migration" id="create org table v1" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.998584935Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:44 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:44.999042658Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.001672749Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.004306928Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.007340896Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.009792307Z level=info msg="Executing migration" id="Update org table charset" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.012567967Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.015349042Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.01825873Z level=info msg="Executing migration" id="create dashboard table" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.020914764Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.023631808Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.026210618Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-24T11:40:45.055 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.029003156Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.03147241Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.034077679Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.036793954Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.039258469Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.042068464Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.044846568Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.047230012Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-24T11:40:45.056 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.050127935Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-24T11:40:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.224+0000 7feb0e5c2700 1 -- 172.21.15.114:0/83070440 >> v1:172.21.15.119:6789/0 conn(0x7feb08108f70 legacy=0x7feb0810b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.224+0000 7feb0e5c2700 1 -- 172.21.15.114:0/83070440 shutdown_connections 2024-03-24T11:40:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.224+0000 7feb0e5c2700 1 -- 172.21.15.114:0/83070440 >> 172.21.15.114:0/83070440 conn(0x7feb08076da0 msgr2=0x7feb080fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.224+0000 7feb0e5c2700 1 -- 172.21.15.114:0/83070440 shutdown_connections 2024-03-24T11:40:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.224+0000 7feb0e5c2700 1 -- 172.21.15.114:0/83070440 wait complete. 2024-03-24T11:40:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb0e5c2700 1 Processor -- start 2024-03-24T11:40:45.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb0e5c2700 1 -- start start 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb0e5c2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feb081a89d0 con 0x7feb081002e0 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb0e5c2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feb081a9b70 con 0x7feb08108f70 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb0e5c2700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feb081aad10 con 0x7feb0810cb10 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb0cb5f700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7feb0810cb10 0x7feb081a72b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33328/0 (socket says 172.21.15.114:33328) 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb0cb5f700 1 -- 172.21.15.114:0/2201235075 learned_addr learned my addr 172.21.15.114:0/2201235075 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb077fe700 1 --1- 172.21.15.114:0/2201235075 >> v1:172.21.15.119:6789/0 conn(0x7feb08108f70 0x7feb08104450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.114:46858/0 (socket says 172.21.15.114:46858) 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135978081 0 0) 0x7feb081a89d0 con 0x7feb081002e0 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf00035b0 con 0x7feb081002e0 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670915637 0 0) 0x7feb081a9b70 con 0x7feb08108f70 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb081a89d0 con 0x7feb08108f70 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255209534 0 0) 0x7feb081aad10 con 0x7feb0810cb10 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.225+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb081a9b70 con 0x7feb0810cb10 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888454238 0 0) 0x7feb081a89d0 con 0x7feb08108f70 2024-03-24T11:40:45.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb081aad10 con 0x7feb08108f70 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2346088041 0 0) 0x7feb081a9b70 con 0x7feb0810cb10 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb081a89d0 con 0x7feb0810cb10 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7feaf80043b0 con 0x7feb08108f70 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7feb00001c80 con 0x7feb0810cb10 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2839821147 0 0) 0x7feb081aad10 con 0x7feb08108f70 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 >> v1:172.21.15.114:6790/0 conn(0x7feb0810cb10 legacy=0x7feb081a72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 >> v1:172.21.15.114:6789/0 conn(0x7feb081002e0 legacy=0x7feb08103d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb081abef0 con 0x7feb08108f70 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb081a9d00 con 0x7feb08108f70 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb081aa180 con 0x7feb08108f70 2024-03-24T11:40:45.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7feaf8004560 con 0x7feb08108f70 2024-03-24T11:40:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.226+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7feaf8004de0 con 0x7feb08108f70 2024-03-24T11:40:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.227+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf4005120 con 0x7feb08108f70 2024-03-24T11:40:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.228+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7feaf8004f60 con 0x7feb08108f70 2024-03-24T11:40:45.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.228+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7feaf8088450 con 0x7feb08108f70 2024-03-24T11:40:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.233+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7feaf8055930 con 0x7feb08108f70 2024-03-24T11:40:45.301 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.052673638Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-24T11:40:45.301 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.055287673Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-24T11:40:45.301 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.057875688Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.060508195Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.063185419Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.065967145Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.068707502Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.071306861Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.072237576Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.072903646Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.073715072Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.074643394Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.075323578Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.076263436Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.077001814Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.077806641Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-24T11:40:45.302 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.078735305Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.079440987Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.080354012Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.08102369Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.081919406Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.082740082Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.083676441Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.084487979Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.085173548Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.086097019Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.086790936Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.087713451Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.088530709Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.089293655Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.090294218Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-24T11:40:45.303 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.091031401Z level=info msg="Executing migration" id="create data_source table" 2024-03-24T11:40:45.304 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.09192602Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-24T11:40:45.304 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.092600658Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-24T11:40:45.304 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.093384781Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-24T11:40:45.304 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.094197343Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T11:40:45.304 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.095168713Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-24T11:40:45.304 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.096321896Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-24T11:40:45.305 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.097189108Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-24T11:40:45.305 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.09790618Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-24T11:40:45.305 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.09868539Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.099437236Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.100253625Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.1011271Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.102119774Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.102819465Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.103778715Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.104519714Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.105311176Z level=info msg="Executing migration" id="Add uid column" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.106164027Z level=info msg="Executing migration" id="Update uid value" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.107186764Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.107930692Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.108772835Z level=info msg="Executing migration" id="create api_key table" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.109765977Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-24T11:40:45.306 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.110503284Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.111481467Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.112287492Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.113168355Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.115914481Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.116383463Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.119281346Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.122139488Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.124804687Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.125264308Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.127865174Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.130663047Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.131122821Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-24T11:40:45.307 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.131428722Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-24T11:40:45.310 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.134366297Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-24T11:40:45.310 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.135136152Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-24T11:40:45.310 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.137186909Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-24T11:40:45.310 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.139908442Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-24T11:40:45.310 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.142528371Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.145197865Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.148112562Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.150901061Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.15367428Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.156202455Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.159017725Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.161391183Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.163901133Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.166570998Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.168862243Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.171504527Z level=info msg="Executing migration" id="create quota table v1" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.174267027Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T11:40:45.311 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.176933629Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.179559816Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.179981577Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.182685502Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.185402625Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.188189555Z level=info msg="Executing migration" id="create session table" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.18865005Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.188942479Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-24T11:40:45.312 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.191449287Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.191885405Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.194728469Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.195151973Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.195436521Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.195694135Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.195974249Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.19743367Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.199781255Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.200555214Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.201186678Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-24T11:40:45.313 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.201792072Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.202748777Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.203201146Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.203612637Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.204571255Z level=info msg="Executing migration" id="create alert table v1" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.20521636Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.206106934Z level=info msg="Executing migration" id="add index alert state" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.206869762Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.207435475Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.207798191Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.20820409Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.208584276Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.209984261Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-24T11:40:45.314 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.210387094Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.210796016Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.21108216Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.211431432Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.211772713Z level=info msg="Executing migration" id="Add column is_default" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.212380495Z level=info msg="Executing migration" id="Add column frequency" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.212996134Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.213657557Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.214266712Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.214677684Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.214931951Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.215236258Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-24T11:40:45.315 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.215615552Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.21602767Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.216407797Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.216756568Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.217129246Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.217741536Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.218365406Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.21867367Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.219079571Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-24T11:40:45.318 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.219481501Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.220080808Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.220372485Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.22075191Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.221140461Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.221437051Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.221866959Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.222274117Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-24T11:40:45.319 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.222673886Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-24T11:40:45.320 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.223080249Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-24T11:40:45.320 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.223509942Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-24T11:40:45.320 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.22434947Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-24T11:40:45.320 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.225151155Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.225833329Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.226692718Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.227476824Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.228301168Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.229095938Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.229955923Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.23156202Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.232349737Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.233185754Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.234098014Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.234883084Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.235865153Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-24T11:40:45.321 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.236758482Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.237620982Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.238461255Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.23932185Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.240100317Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.240922243Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.241954489Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.242678027Z level=info msg="Executing migration" id="Move region to single row" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.243484602Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-24T11:40:45.322 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.243910704Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.244369249Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.245249737Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.24624002Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.246947355Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.249828808Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.252515246Z level=info msg="Executing migration" id="create test_data table" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.252923617Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.25375767Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.254564193Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.255326053Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.258108208Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.260823676Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-24T11:40:45.323 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.263286321Z level=info msg="Executing migration" id="create team table" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.26594299Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.268830621Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.271589007Z level=info msg="Executing migration" id="create team member table" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.274235345Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.276912246Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.279647483Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.282598661Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.285116162Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-24T11:40:45.324 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.287793962Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-24T11:40:45.325 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.290559251Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-24T11:40:45.325 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.293202803Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-24T11:40:45.325 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.295741537Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T11:40:45.325 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.298752354Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T11:40:45.325 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.301355767Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-24T11:40:45.325 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.304074921Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-24T11:40:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.458+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7feaf40053d0 con 0x7feb08108f70 2024-03-24T11:40:45.459 INFO:teuthology.orchestra.run.smithi114.stdout:236223201300 2024-03-24T11:40:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.459+0000 7feb057fa700 1 -- 172.21.15.114:0/2201235075 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 3646229204) 0x7feaf8053ca0 con 0x7feb08108f70 2024-03-24T11:40:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.462+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 >> v1:172.21.15.114:6800/1253397276 conn(0x7feaf006e5b0 legacy=0x7feaf0070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.462+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 >> v1:172.21.15.119:6789/0 conn(0x7feb08108f70 legacy=0x7feb08104450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.462+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 shutdown_connections 2024-03-24T11:40:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.462+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 >> 172.21.15.114:0/2201235075 conn(0x7feb08076da0 msgr2=0x7feb081085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.462+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 shutdown_connections 2024-03-24T11:40:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.462+0000 7feb0e5c2700 1 -- 172.21.15.114:0/2201235075 wait complete. 2024-03-24T11:40:45.552 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.30650409Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-24T11:40:45.552 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.307003026Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-24T11:40:45.552 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.309514219Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-24T11:40:45.552 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.312011263Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-24T11:40:45.552 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.314736208Z level=info msg="Executing migration" id="create tag table" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.315133764Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.317721646Z level=info msg="Executing migration" id="create login attempt table" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.320091402Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.322656793Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.323255529Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.326023193Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.328501554Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.331043802Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.331525059Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.334230463Z level=info msg="Executing migration" id="create user auth table" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.334633044Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.336810746Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.337644443Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-24T11:40:45.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.338538034Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-24T11:40:45.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.339312769Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-24T11:40:45.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.340141621Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-24T11:40:45.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.340955688Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-24T11:40:45.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.341380998Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-24T11:40:45.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.342157969Z level=info msg="Executing migration" id="create server_lock table" 2024-03-24T11:40:45.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.343122883Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.343724248Z level=info msg="Executing migration" id="create user auth token table" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.344137195Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.344958536Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.345717329Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.346519628Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.347317431Z level=info msg="Executing migration" id="create cache_data table" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.348322249Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.348974625Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.349797415Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.350230576Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.351086545Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.35184204Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.352648331Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.353073752Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-24T11:40:45.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.353991601Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.354730122Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.35551999Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.35593767Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.356355462Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.357177035Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.357968274Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.358753329Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.359043231Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.359907812Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.360861405Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.361496802Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.361784148Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.364498413Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.364914193Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T11:40:45.556 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.367322752Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.370124967Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.370910886Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.37381582Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.376500136Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.380030187Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.383313658Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.386106914Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.388714463Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.391717867Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.392228222Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.395069778Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.397838576Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.400244728Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.400560376Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-24T11:40:45.557 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.403242906Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-24T11:40:45.558 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.405814528Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-24T11:40:45.558 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.408177803Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T11:40:45.558 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.410800421Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.413192297Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.415960333Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.418621434Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.419141491Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.421769738Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.424563803Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.427088118Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.427533706Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.430089822Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T11:40:45.559 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.432647816Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.43505654Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.435954197Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.438532173Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.440980696Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.441868873Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.442739332Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.443611069Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.44435721Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.445242763Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.445559269Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.446480454Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.447229741Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.448146664Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.448574269Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.4494061Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-24T11:40:45.560 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.450282527Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.451089045Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.451538542Z level=info msg="Executing migration" id="create alert_image table" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.452322919Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.453158814Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.45395766Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.454389114Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.455291532Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.455509637Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.456036383Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.456852445Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.457310471Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.45744679Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.457581902Z level=info msg="alerts found to migrate" alerts=0 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.460119593Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.462599175Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-24T11:40:45.561 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.46559882Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.466173336Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.46920517Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.471903979Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.474607015Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.477338388Z level=info msg="Executing migration" id="create data_keys table" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.480212061Z level=info msg="Executing migration" id="create secrets table" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.483073445Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.487393369Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.490259859Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.4930367Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.497104171Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.501361054Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.504215418Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-24T11:40:45.562 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.506954472Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.509516186Z level=info msg="Executing migration" id="create permission table" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.509926913Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.51282315Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.51523918Z level=info msg="Executing migration" id="create role table" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.517840293Z level=info msg="Executing migration" id="add column display_name" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.5189229Z level=info msg="Executing migration" id="add column group_name" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.52157748Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.524361829Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.527016651Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.527662263Z level=info msg="Executing migration" id="create team role table" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.530551976Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.533083171Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.535492387Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.535960617Z level=info msg="Executing migration" id="create user role table" 2024-03-24T11:40:45.563 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.536787475Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.537608306Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.538531761Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.53918279Z level=info msg="Executing migration" id="create builtin role table" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.539989308Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.540779955Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.54165032Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.542682676Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.543217933Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.54400063Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.544419037Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.54482001Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.545274168Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.545736984Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.546842253Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.54729096Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-24T11:40:45.564 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.54774384Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.548013052Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.548462505Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.548920523Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.549309045Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.549661579Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.550003755Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.550414676Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.550851711Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.551855534Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.552270634Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.552823961Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.55343335Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.554007808Z level=info msg="Executing migration" id="add correlation config column" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.555556562Z level=info msg="Executing migration" id="create entity_events table" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.55613409Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-24T11:40:45.565 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.556914636Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:40:45.566 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.557029342Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:40:45.566 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.557447062Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:40:45.566 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.557547391Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:40:45.566 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.55783915Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-24T11:40:45.566 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.558306689Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.558721266Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.559164627Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.559619415Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.560016806Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.560480366Z level=info msg="Executing migration" id="Drop public config table" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.560884789Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.561323924Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.561821732Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:40:45.805 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.562293547Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.562734874Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.565788576Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.56681412Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.567846275Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.568177734Z level=info msg="Executing migration" id="add share column" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.569209816Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.569673602Z level=info msg="Executing migration" id="create file table" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.570299591Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-24T11:40:45.806 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.570764878Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.571258468Z level=info msg="Executing migration" id="create file_meta table" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.5722491Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.572910544Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.573224959Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.574168766Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.574963188Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.57583704Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.576928261Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.577811819Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.578775109Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.579497767Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.579845538Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-24T11:40:45.807 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.580626475Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.581675444Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.582718079Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.583204429Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.592944025Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.593810666Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.594809061Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.598085748Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.598962927Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.599877017Z level=info msg="Executing migration" id="create folder table" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.602654328Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.603119742Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.605945568Z level=info msg="Executing migration" id="Update folder title length" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.608757642Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=migrator t=2024-03-24T11:40:45.611248201Z level=info msg="migrations completed" performed=485 skipped=0 duration=651.687809ms 2024-03-24T11:40:45.808 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=sqlstore t=2024-03-24T11:40:45.611685849Z level=info msg="Created default organization" 2024-03-24T11:40:45.809 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=plugin.loader t=2024-03-24T11:40:45.666061051Z level=info msg="Plugin registered" pluginID=input 2024-03-24T11:40:45.809 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=plugin.loader t=2024-03-24T11:40:45.693032509Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-24T11:40:45.809 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=plugin.loader t=2024-03-24T11:40:45.693056005Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-24T11:40:45.809 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=secrets t=2024-03-24T11:40:45.693113521Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-24T11:40:45.809 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=query_data t=2024-03-24T11:40:45.696482456Z level=info msg="Query Service initialization" 2024-03-24T11:40:45.809 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=live.push_http t=2024-03-24T11:40:45.699229823Z level=info msg="Live Push Gateway initialization" 2024-03-24T11:40:45.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.540+0000 7f0da1f49700 1 -- 172.21.15.114:0/127394144 >> v1:172.21.15.114:6790/0 conn(0x7f0d9c10a8e0 legacy=0x7f0d9c10ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.540+0000 7f0da1f49700 1 -- 172.21.15.114:0/127394144 shutdown_connections 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.540+0000 7f0da1f49700 1 -- 172.21.15.114:0/127394144 >> 172.21.15.114:0/127394144 conn(0x7f0d9c0fd560 msgr2=0x7f0d9c0ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.540+0000 7f0da1f49700 1 -- 172.21.15.114:0/127394144 shutdown_connections 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.540+0000 7f0da1f49700 1 -- 172.21.15.114:0/127394144 wait complete. 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.541+0000 7f0da1f49700 1 Processor -- start 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.541+0000 7f0da1f49700 1 -- start start 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.541+0000 7f0da1f49700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d9c1a68d0 con 0x7f0d9c107a30 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.541+0000 7f0da1f49700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d9c1a7ab0 con 0x7f0d9c10a8e0 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.541+0000 7f0da1f49700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d9c1a8cb0 con 0x7f0d9c10e4b0 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d9b7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0d9c107a30 0x7f0d9c10db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43466/0 (socket says 172.21.15.114:43466) 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d9b7fe700 1 -- 172.21.15.114:0/3579484245 learned_addr learned my addr 172.21.15.114:0/3579484245 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20568573 0 0) 0x7f0d9c1a8cb0 con 0x7f0d9c10e4b0 2024-03-24T11:40:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d840035b0 con 0x7f0d9c10e4b0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893524396 0 0) 0x7f0d9c1a68d0 con 0x7f0d9c107a30 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c1a8cb0 con 0x7f0d9c107a30 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854109892 0 0) 0x7f0d9c1a7ab0 con 0x7f0d9c10a8e0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c1a68d0 con 0x7f0d9c10a8e0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610305920 0 0) 0x7f0d9c1a8cb0 con 0x7f0d9c107a30 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d9c1a7ab0 con 0x7f0d9c107a30 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1767644766 0 0) 0x7f0d840035b0 con 0x7f0d9c10e4b0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d9c1a8cb0 con 0x7f0d9c10e4b0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0d8c0012d0 con 0x7f0d9c107a30 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426437225 0 0) 0x7f0d9c1a68d0 con 0x7f0d9c10a8e0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d840035b0 con 0x7f0d9c10a8e0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0d94001bb0 con 0x7f0d9c10e4b0 2024-03-24T11:40:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0d90004320 con 0x7f0d9c10a8e0 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3783553171 0 0) 0x7f0d9c1a7ab0 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 >> v1:172.21.15.114:6790/0 conn(0x7f0d9c10e4b0 legacy=0x7f0d9c1a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 >> v1:172.21.15.119:6789/0 conn(0x7f0d9c10a8e0 legacy=0x7f0d9c1a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c1a9eb0 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9c1a8ea0 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.542+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9c1a9320 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.543+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f0d8c003b10 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.543+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f0d8c005b00 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.543+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d88005120 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.544+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f0d8c003680 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.544+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f0d8c0882a0 con 0x7f0d9c107a30 2024-03-24T11:40:45.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.549+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f0d8c0557e0 con 0x7f0d9c107a30 2024-03-24T11:40:45.894 INFO:teuthology.orchestra.run.smithi114.stdout:55834574898 2024-03-24T11:40:45.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.751+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0d880053d0 con 0x7f0d9c107a30 2024-03-24T11:40:45.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.751+0000 7f0d98ff9700 1 -- 172.21.15.114:0/3579484245 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 2868210388) 0x7f0d8c053b50 con 0x7f0d9c107a30 2024-03-24T11:40:45.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.754+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 >> v1:172.21.15.114:6800/1253397276 conn(0x7f0d8406e660 legacy=0x7f0d84070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.754+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 >> v1:172.21.15.114:6789/0 conn(0x7f0d9c107a30 legacy=0x7f0d9c10db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:45.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.754+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 shutdown_connections 2024-03-24T11:40:45.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.754+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 >> 172.21.15.114:0/3579484245 conn(0x7f0d9c0fd560 msgr2=0x7f0d9c1119b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:45.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.754+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 shutdown_connections 2024-03-24T11:40:45.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:45.754+0000 7f0da1f49700 1 -- 172.21.15.114:0/3579484245 wait complete. 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: Deploying daemon node-exporter.a on smithi114 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2201235075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3579484245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: Deploying daemon node-exporter.a on smithi114 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2201235075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:46.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3579484245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: Deploying daemon node-exporter.a on smithi114 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2201235075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:46.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3579484245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:40:46.489 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:46 smithi114 systemd[1]: Starting Ceph node-exporter.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:40:46.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=infra.usagestats.collector t=2024-03-24T11:40:46.301248525Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-24T11:40:46.554 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=provisioning.datasources t=2024-03-24T11:40:46.302452475Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=provisioning.datasources t=2024-03-24T11:40:46.31348548Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=provisioning.alerting t=2024-03-24T11:40:46.324652182Z level=info msg="starting to provision alerting" 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=provisioning.alerting t=2024-03-24T11:40:46.32467159Z level=info msg="finished to provision alerting" 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=grafanaStorageLogger t=2024-03-24T11:40:46.32488888Z level=info msg="storage starting" 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=ngalert.state.manager t=2024-03-24T11:40:46.324926613Z level=info msg="Warming state cache for startup" 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=http.server t=2024-03-24T11:40:46.326636602Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=ngalert.state.manager t=2024-03-24T11:40:46.337904911Z level=info msg="State cache has been initialized" states=0 duration=12.977272ms 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=ticker t=2024-03-24T11:40:46.337971827Z level=info msg=starting first_tick=2024-03-24T11:40:50Z 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=ngalert.multiorg.alertmanager t=2024-03-24T11:40:46.337963179Z level=info msg="starting MultiOrg Alertmanager" 2024-03-24T11:40:46.555 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=sqlstore.transactions t=2024-03-24T11:40:46.353911462Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:40:47.056 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574898 for osd.1 2024-03-24T11:40:47.056 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:46 smithi114 ceph-mon[30312]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:47.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:46 smithi114 ceph-mon[42466]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:47.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:46 smithi119 ceph-mon[33714]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:47.810 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201296 got 236223201300 for osd.7 2024-03-24T11:40:47.810 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:48.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.302+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/841247686 >> v1:172.21.15.114:6790/0 conn(0x7f9a2810e4a0 legacy=0x7f9a28110960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.302+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/841247686 shutdown_connections 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.302+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/841247686 >> 172.21.15.114:0/841247686 conn(0x7f9a280fd550 msgr2=0x7f9a280ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.302+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/841247686 shutdown_connections 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.302+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/841247686 wait complete. 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2f6d4700 1 Processor -- start 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2f6d4700 1 -- start start 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2f6d4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a281a8a90 con 0x7f9a2810a8d0 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2f6d4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a281a9c70 con 0x7f9a2810e4a0 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2f6d4700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a281aae70 con 0x7f9a28107a20 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2d470700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9a28107a20 0x7f9a28075820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33372/0 (socket says 172.21.15.114:33372) 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2cc6f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9a2810a8d0 0x7f9a281a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43492/0 (socket says 172.21.15.114:43492) 2024-03-24T11:40:48.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a2d470700 1 -- 172.21.15.114:0/2838817933 learned_addr learned my addr 172.21.15.114:0/2838817933 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:48.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.303+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590988880 0 0) 0x7f9a281aae70 con 0x7f9a28107a20 2024-03-24T11:40:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a140035b0 con 0x7f9a28107a20 2024-03-24T11:40:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602439784 0 0) 0x7f9a281a8a90 con 0x7f9a2810a8d0 2024-03-24T11:40:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a281aae70 con 0x7f9a2810a8d0 2024-03-24T11:40:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717459653 0 0) 0x7f9a281a9c70 con 0x7f9a2810e4a0 2024-03-24T11:40:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a281a8a90 con 0x7f9a2810e4a0 2024-03-24T11:40:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2223700997 0 0) 0x7f9a140035b0 con 0x7f9a28107a20 2024-03-24T11:40:48.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a281a9c70 con 0x7f9a28107a20 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9a18001930 con 0x7f9a28107a20 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3701752195 0 0) 0x7f9a281a8a90 con 0x7f9a2810e4a0 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a140035b0 con 0x7f9a2810e4a0 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265772123 0 0) 0x7f9a281aae70 con 0x7f9a2810a8d0 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a281a8a90 con 0x7f9a2810a8d0 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9a24001630 con 0x7f9a2810a8d0 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9a20004670 con 0x7f9a2810e4a0 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4044317963 0 0) 0x7f9a281a9c70 con 0x7f9a28107a20 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 >> v1:172.21.15.119:6789/0 conn(0x7f9a2810e4a0 legacy=0x7f9a281a7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 >> v1:172.21.15.114:6789/0 conn(0x7f9a2810a8d0 legacy=0x7f9a281a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a281ac070 con 0x7f9a28107a20 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a281a8c80 con 0x7f9a28107a20 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a281a9100 con 0x7f9a28107a20 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f9a18001cc0 con 0x7f9a28107a20 2024-03-24T11:40:48.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.304+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9a18005cb0 con 0x7f9a28107a20 2024-03-24T11:40:48.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.305+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0c005120 con 0x7f9a28107a20 2024-03-24T11:40:48.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.306+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f9a18004070 con 0x7f9a28107a20 2024-03-24T11:40:48.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.306+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f9a180883b0 con 0x7f9a28107a20 2024-03-24T11:40:48.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.311+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f9a180559d0 con 0x7f9a28107a20 2024-03-24T11:40:48.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.527+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9a0c0053d0 con 0x7f9a28107a20 2024-03-24T11:40:48.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.528+0000 7f9a1e7fc700 1 -- 172.21.15.114:0/2838817933 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 4006577385) 0x7f9a18053d40 con 0x7f9a28107a20 2024-03-24T11:40:48.528 INFO:teuthology.orchestra.run.smithi114.stdout:111669149738 2024-03-24T11:40:48.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.530+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 >> v1:172.21.15.114:6800/1253397276 conn(0x7f9a1406e600 legacy=0x7f9a14070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:48.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.530+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 >> v1:172.21.15.114:6790/0 conn(0x7f9a28107a20 legacy=0x7f9a28075820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:48.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.530+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 shutdown_connections 2024-03-24T11:40:48.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.530+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 >> 172.21.15.114:0/2838817933 conn(0x7f9a280fd550 msgr2=0x7f9a28100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:48.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.530+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 shutdown_connections 2024-03-24T11:40:48.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:48.530+0000 7f9a2f6d4700 1 -- 172.21.15.114:0/2838817933 wait complete. 2024-03-24T11:40:48.739 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:48 smithi114 bash[79788]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T11:40:49.239 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:48 smithi114 bash[79788]: Getting image source signatures 2024-03-24T11:40:49.239 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:48 smithi114 bash[79788]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T11:40:49.239 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:48 smithi114 bash[79788]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T11:40:49.239 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:48 smithi114 bash[79788]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T11:40:49.279 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149734 got 111669149738 for osd.3 2024-03-24T11:40:49.279 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:49.279 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:40:49.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:40:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:49 smithi114 ceph-mon[42466]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2838817933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:40:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:49 smithi114 ceph-mon[30312]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2838817933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:40:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:49 smithi119 ceph-mon[33714]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2838817933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:40:50.242 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:50.615 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:50 smithi114 bash[79788]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:40:50.989 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:50 smithi114 bash[79788]: Writing manifest to image destination 2024-03-24T11:40:51.425 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 podman[79788]: 2024-03-24 11:40:51.209866085 +0000 UTC m=+2.826157056 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:40:51.425 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 podman[79788]: 2024-03-24 11:40:51.425161794 +0000 UTC m=+3.041452762 container create 13608fac3d0d4cb64423af21d8313fe985db35a2d0ea30a1c9b7c8f7e8a033e8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:51.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-mon[30312]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:51.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-mon[42466]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:51.676 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 podman[79788]: 2024-03-24 11:40:51.669635911 +0000 UTC m=+3.285926878 container init 13608fac3d0d4cb64423af21d8313fe985db35a2d0ea30a1c9b7c8f7e8a033e8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:51.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:51 smithi119 ceph-mon[33714]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:51.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:40:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:40:51.989 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 podman[79788]: 2024-03-24 11:40:51.676173389 +0000 UTC m=+3.292464351 container start 13608fac3d0d4cb64423af21d8313fe985db35a2d0ea30a1c9b7c8f7e8a033e8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:51.989 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.680Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.680Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.681Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.681Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.681Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.681Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:40:51.990 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:40:51.991 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:40:51.992 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:40:51.993 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:40:51.994 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:40:51.994 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a[79971]: ts=2024-03-24T11:40:51.682Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:40:51.994 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 bash[79788]: 13608fac3d0d4cb64423af21d8313fe985db35a2d0ea30a1c9b7c8f7e8a033e8 2024-03-24T11:40:51.994 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 11:40:51 smithi114 systemd[1]: Started Ceph node-exporter.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:40:52.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:52.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:52.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.606 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.606 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:52 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:52 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:52 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:52 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:52 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.846+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/2394252231 >> v1:172.21.15.114:6790/0 conn(0x7f8e64107a10 legacy=0x7f8e64107df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.846+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/2394252231 shutdown_connections 2024-03-24T11:40:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.846+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/2394252231 >> 172.21.15.114:0/2394252231 conn(0x7f8e640fd520 msgr2=0x7f8e640ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.846+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/2394252231 shutdown_connections 2024-03-24T11:40:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.846+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/2394252231 wait complete. 2024-03-24T11:40:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e6b5f5700 1 Processor -- start 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e6b5f5700 1 -- start start 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e6b5f5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e6410dfc0 con 0x7f8e6410a8c0 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e6b5f5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e6410e190 con 0x7f8e6410e490 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e6b5f5700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e641a8fb0 con 0x7f8e64107a10 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e69391700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f8e64107a10 0x7f8e6410afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39370/0 (socket says 172.21.15.114:39370) 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e68b90700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8e6410a8c0 0x7f8e6410b6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39054/0 (socket says 172.21.15.114:39054) 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e69391700 1 -- 172.21.15.114:0/1857313308 learned_addr learned my addr 172.21.15.114:0/1857313308 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203435567 0 0) 0x7f8e6410dfc0 con 0x7f8e6410a8c0 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e500035b0 con 0x7f8e6410a8c0 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637905950 0 0) 0x7f8e641a8fb0 con 0x7f8e64107a10 2024-03-24T11:40:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e6410dfc0 con 0x7f8e64107a10 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545285680 0 0) 0x7f8e6410e190 con 0x7f8e6410e490 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e641a8fb0 con 0x7f8e6410e490 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2147105593 0 0) 0x7f8e6410dfc0 con 0x7f8e64107a10 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e6410e190 con 0x7f8e64107a10 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.847+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f8e5c003680 con 0x7f8e64107a10 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2572960244 0 0) 0x7f8e500035b0 con 0x7f8e6410a8c0 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e6410dfc0 con 0x7f8e6410a8c0 2024-03-24T11:40:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716830998 0 0) 0x7f8e641a8fb0 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e500035b0 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f8e54001830 con 0x7f8e6410a8c0 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f8e60001bd0 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 937633367 0 0) 0x7f8e500035b0 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 >> v1:172.21.15.114:6790/0 conn(0x7f8e64107a10 legacy=0x7f8e6410afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 >> v1:172.21.15.114:6789/0 conn(0x7f8e6410a8c0 legacy=0x7f8e6410b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e641aa150 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e641a91a0 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f8e600020a0 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e641a9620 con 0x7f8e6410e490 2024-03-24T11:40:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.848+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f8e60004b90 con 0x7f8e6410e490 2024-03-24T11:40:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.849+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e64103150 con 0x7f8e6410e490 2024-03-24T11:40:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.849+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f8e60003a20 con 0x7f8e6410e490 2024-03-24T11:40:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.850+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f8e60087480 con 0x7f8e6410e490 2024-03-24T11:40:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:52.856+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f8e60054960 con 0x7f8e6410e490 2024-03-24T11:40:53.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.045+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 --> v1:172.21.15.114:6800/1253397276 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e640685d0 con 0x7f8e5006e5b0 2024-03-24T11:40:53.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.054+0000 7f8e5a7fc700 1 -- 172.21.15.114:0/1857313308 <== mgr.14589 v1:172.21.15.114:6800/1253397276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346475 (unknown 2965378022 0 1742038151) 0x7f8e640685d0 con 0x7f8e5006e5b0 2024-03-24T11:40:53.060 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:40:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.057+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 >> v1:172.21.15.114:6800/1253397276 conn(0x7f8e5006e5b0 legacy=0x7f8e50070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.057+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 >> v1:172.21.15.119:6789/0 conn(0x7f8e6410e490 legacy=0x7f8e641a5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.057+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 shutdown_connections 2024-03-24T11:40:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.057+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 >> 172.21.15.114:0/1857313308 conn(0x7f8e640fd520 msgr2=0x7f8e6410f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.057+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 shutdown_connections 2024-03-24T11:40:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:53.057+0000 7f8e6b5f5700 1 -- 172.21.15.114:0/1857313308 wait complete. 2024-03-24T11:40:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-24T11:40:53.304 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:53 smithi119 bash[66701]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T11:40:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:53 smithi114 ceph-mon[30312]: Deploying daemon node-exporter.b on smithi119 2024-03-24T11:40:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:53 smithi114 ceph-mon[30312]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:53 smithi114 ceph-mon[42466]: Deploying daemon node-exporter.b on smithi119 2024-03-24T11:40:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:53 smithi114 ceph-mon[42466]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:53 smithi119 ceph-mon[33714]: Deploying daemon node-exporter.b on smithi119 2024-03-24T11:40:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:53 smithi119 ceph-mon[33714]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:53.804 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:53 smithi119 bash[66701]: Getting image source signatures 2024-03-24T11:40:53.804 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:53 smithi119 bash[66701]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T11:40:53.804 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:53 smithi119 bash[66701]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T11:40:53.805 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:53 smithi119 bash[66701]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T11:40:53.913 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-24T11:40:52.395636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1016,"num_read_kb":862,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235992,"kb_used_data":7356,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725224,"statfs":{"total":767960285184,"available":767718629376,"internally_reserved":0,"allocated":7532544,"data_stored":4008986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002711"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376185+0000","last_change":"2024-03-24T11:39:28.352146+0000","last_active":"2024-03-24T11:40:00.376185+0000","last_peered":"2024-03-24T11:40:00.376185+0000","last_clean":"2024-03-24T11:40:00.376185+0000","last_became_active":"2024-03-24T11:39:28.351793+0000","last_became_peered":"2024-03-24T11:39:28.351793+0000","last_unstale":"2024-03-24T11:40:00.376185+0000","last_undegraded":"2024-03-24T11:40:00.376185+0000","last_fullsized":"2024-03-24T11:40:00.376185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:29.116830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214115+0000","last_change":"2024-03-24T11:39:26.341966+0000","last_active":"2024-03-24T11:40:01.214115+0000","last_peered":"2024-03-24T11:40:01.214115+0000","last_clean":"2024-03-24T11:40:01.214115+0000","last_became_active":"2024-03-24T11:39:26.341063+0000","last_became_peered":"2024-03-24T11:39:26.341063+0000","last_unstale":"2024-03-24T11:40:01.214115+0000","last_undegraded":"2024-03-24T11:40:01.214115+0000","last_fullsized":"2024-03-24T11:40:01.214115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:39.315934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215080+0000","last_change":"2024-03-24T11:39:22.275045+0000","last_active":"2024-03-24T11:40:01.215080+0000","last_peered":"2024-03-24T11:40:01.215080+0000","last_clean":"2024-03-24T11:40:01.215080+0000","last_became_active":"2024-03-24T11:39:22.274707+0000","last_became_peered":"2024-03-24T11:39:22.274707+0000","last_unstale":"2024-03-24T11:40:01.215080+0000","last_undegraded":"2024-03-24T11:40:01.215080+0000","last_fullsized":"2024-03-24T11:40:01.215080+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:33.849147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375798+0000","last_change":"2024-03-24T11:39:48.132081+0000","last_active":"2024-03-24T11:40:00.375798+0000","last_peered":"2024-03-24T11:40:00.375798+0000","last_clean":"2024-03-24T11:40:00.375798+0000","last_became_active":"2024-03-24T11:39:48.126605+0000","last_became_peered":"2024-03-24T11:39:48.126605+0000","last_unstale":"2024-03-24T11:40:00.375798+0000","last_undegraded":"2024-03-24T11:40:00.375798+0000","last_fullsized":"2024-03-24T11:40:00.375798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:57:29.208780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373497+0000","last_change":"2024-03-24T11:39:29.121106+0000","last_active":"2024-03-24T11:40:00.373497+0000","last_peered":"2024-03-24T11:40:00.373497+0000","last_clean":"2024-03-24T11:40:00.373497+0000","last_became_active":"2024-03-24T11:39:29.120714+0000","last_became_peered":"2024-03-24T11:39:29.120714+0000","last_unstale":"2024-03-24T11:40:00.373497+0000","last_undegraded":"2024-03-24T11:40:00.373497+0000","last_fullsized":"2024-03-24T11:40:00.373497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:18.037707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375800+0000","last_change":"2024-03-24T11:39:26.342934+0000","last_active":"2024-03-24T11:40:00.375800+0000","last_peered":"2024-03-24T11:40:00.375800+0000","last_clean":"2024-03-24T11:40:00.375800+0000","last_became_active":"2024-03-24T11:39:26.341799+0000","last_became_peered":"2024-03-24T11:39:26.341799+0000","last_unstale":"2024-03-24T11:40:00.375800+0000","last_undegraded":"2024-03-24T11:40:00.375800+0000","last_fullsized":"2024-03-24T11:40:00.375800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:26:13.575390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374566+0000","last_change":"2024-03-24T11:39:22.269836+0000","last_active":"2024-03-24T11:40:00.374566+0000","last_peered":"2024-03-24T11:40:00.374566+0000","last_clean":"2024-03-24T11:40:00.374566+0000","last_became_active":"2024-03-24T11:39:22.269554+0000","last_became_peered":"2024-03-24T11:39:22.269554+0000","last_unstale":"2024-03-24T11:40:00.374566+0000","last_undegraded":"2024-03-24T11:40:00.374566+0000","last_fullsized":"2024-03-24T11:40:00.374566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:42:50.024642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375767+0000","last_change":"2024-03-24T11:39:24.325168+0000","last_active":"2024-03-24T11:40:00.375767+0000","last_peered":"2024-03-24T11:40:00.375767+0000","last_clean":"2024-03-24T11:40:00.375767+0000","last_became_active":"2024-03-24T11:39:24.323915+0000","last_became_peered":"2024-03-24T11:39:24.323915+0000","last_unstale":"2024-03-24T11:40:00.375767+0000","last_undegraded":"2024-03-24T11:40:00.375767+0000","last_fullsized":"2024-03-24T11:40:00.375767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:33.203268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214252+0000","last_change":"2024-03-24T11:39:28.359406+0000","last_active":"2024-03-24T11:40:01.214252+0000","last_peered":"2024-03-24T11:40:01.214252+0000","last_clean":"2024-03-24T11:40:01.214252+0000","last_became_active":"2024-03-24T11:39:28.359087+0000","last_became_peered":"2024-03-24T11:39:28.359087+0000","last_unstale":"2024-03-24T11:40:01.214252+0000","last_undegraded":"2024-03-24T11:40:01.214252+0000","last_fullsized":"2024-03-24T11:40:01.214252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:22.414364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482266+0000","last_change":"2024-03-24T11:39:26.346456+0000","last_active":"2024-03-24T11:40:00.482266+0000","last_peered":"2024-03-24T11:40:00.482266+0000","last_clean":"2024-03-24T11:40:00.482266+0000","last_became_active":"2024-03-24T11:39:26.346107+0000","last_became_peered":"2024-03-24T11:39:26.346107+0000","last_unstale":"2024-03-24T11:40:00.482266+0000","last_undegraded":"2024-03-24T11:40:00.482266+0000","last_fullsized":"2024-03-24T11:40:00.482266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:44.732570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482428+0000","last_change":"2024-03-24T11:39:22.270151+0000","last_active":"2024-03-24T11:40:00.482428+0000","last_peered":"2024-03-24T11:40:00.482428+0000","last_clean":"2024-03-24T11:40:00.482428+0000","last_became_active":"2024-03-24T11:39:22.269862+0000","last_became_peered":"2024-03-24T11:39:22.269862+0000","last_unstale":"2024-03-24T11:40:00.482428+0000","last_undegraded":"2024-03-24T11:40:00.482428+0000","last_fullsized":"2024-03-24T11:40:00.482428+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:53.761052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376412+0000","last_change":"2024-03-24T11:39:24.323561+0000","last_active":"2024-03-24T11:40:00.376412+0000","last_peered":"2024-03-24T11:40:00.376412+0000","last_clean":"2024-03-24T11:40:00.376412+0000","last_became_active":"2024-03-24T11:39:24.323091+0000","last_became_peered":"2024-03-24T11:39:24.323091+0000","last_unstale":"2024-03-24T11:40:00.376412+0000","last_undegraded":"2024-03-24T11:40:00.376412+0000","last_fullsized":"2024-03-24T11:40:00.376412+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:07.730292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482080+0000","last_change":"2024-03-24T11:39:28.352566+0000","last_active":"2024-03-24T11:40:00.482080+0000","last_peered":"2024-03-24T11:40:00.482080+0000","last_clean":"2024-03-24T11:40:00.482080+0000","last_became_active":"2024-03-24T11:39:28.352263+0000","last_became_peered":"2024-03-24T11:39:28.352263+0000","last_unstale":"2024-03-24T11:40:00.482080+0000","last_undegraded":"2024-03-24T11:40:00.482080+0000","last_fullsized":"2024-03-24T11:40:00.482080+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:01:56.522077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497816+0000","last_change":"2024-03-24T11:39:26.341841+0000","last_active":"2024-03-24T11:40:00.497816+0000","last_peered":"2024-03-24T11:40:00.497816+0000","last_clean":"2024-03-24T11:40:00.497816+0000","last_became_active":"2024-03-24T11:39:26.341511+0000","last_became_peered":"2024-03-24T11:39:26.341511+0000","last_unstale":"2024-03-24T11:40:00.497816+0000","last_undegraded":"2024-03-24T11:40:00.497816+0000","last_fullsized":"2024-03-24T11:40:00.497816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:23.921201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482379+0000","last_change":"2024-03-24T11:39:22.276917+0000","last_active":"2024-03-24T11:40:00.482379+0000","last_peered":"2024-03-24T11:40:00.482379+0000","last_clean":"2024-03-24T11:40:00.482379+0000","last_became_active":"2024-03-24T11:39:22.276624+0000","last_became_peered":"2024-03-24T11:39:22.276624+0000","last_unstale":"2024-03-24T11:40:00.482379+0000","last_undegraded":"2024-03-24T11:40:00.482379+0000","last_fullsized":"2024-03-24T11:40:00.482379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:26:48.692979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376417+0000","last_change":"2024-03-24T11:39:24.327759+0000","last_active":"2024-03-24T11:40:00.376417+0000","last_peered":"2024-03-24T11:40:00.376417+0000","last_clean":"2024-03-24T11:40:00.376417+0000","last_became_active":"2024-03-24T11:39:24.327423+0000","last_became_peered":"2024-03-24T11:39:24.327423+0000","last_unstale":"2024-03-24T11:40:00.376417+0000","last_undegraded":"2024-03-24T11:40:00.376417+0000","last_fullsized":"2024-03-24T11:40:00.376417+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:17.824303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377932+0000","last_change":"2024-03-24T11:39:29.120810+0000","last_active":"2024-03-24T11:40:00.377932+0000","last_peered":"2024-03-24T11:40:00.377932+0000","last_clean":"2024-03-24T11:40:00.377932+0000","last_became_active":"2024-03-24T11:39:29.120510+0000","last_became_peered":"2024-03-24T11:39:29.120510+0000","last_unstale":"2024-03-24T11:40:00.377932+0000","last_undegraded":"2024-03-24T11:40:00.377932+0000","last_fullsized":"2024-03-24T11:40:00.377932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:35.044401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215120+0000","last_change":"2024-03-24T11:39:26.342978+0000","last_active":"2024-03-24T11:40:01.215120+0000","last_peered":"2024-03-24T11:40:01.215120+0000","last_clean":"2024-03-24T11:40:01.215120+0000","last_became_active":"2024-03-24T11:39:26.342201+0000","last_became_peered":"2024-03-24T11:39:26.342201+0000","last_unstale":"2024-03-24T11:40:01.215120+0000","last_undegraded":"2024-03-24T11:40:01.215120+0000","last_fullsized":"2024-03-24T11:40:01.215120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:25.363262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.216083+0000","last_change":"2024-03-24T11:39:22.280482+0000","last_active":"2024-03-24T11:40:01.216083+0000","last_peered":"2024-03-24T11:40:01.216083+0000","last_clean":"2024-03-24T11:40:01.216083+0000","last_became_active":"2024-03-24T11:39:22.279980+0000","last_became_peered":"2024-03-24T11:39:22.279980+0000","last_unstale":"2024-03-24T11:40:01.216083+0000","last_undegraded":"2024-03-24T11:40:01.216083+0000","last_fullsized":"2024-03-24T11:40:01.216083+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:35:49.140830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373560+0000","last_change":"2024-03-24T11:39:24.328144+0000","last_active":"2024-03-24T11:40:00.373560+0000","last_peered":"2024-03-24T11:40:00.373560+0000","last_clean":"2024-03-24T11:40:00.373560+0000","last_became_active":"2024-03-24T11:39:24.327800+0000","last_became_peered":"2024-03-24T11:39:24.327800+0000","last_unstale":"2024-03-24T11:40:00.373560+0000","last_undegraded":"2024-03-24T11:40:00.373560+0000","last_fullsized":"2024-03-24T11:40:00.373560+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:05.892896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376313+0000","last_change":"2024-03-24T11:39:29.123532+0000","last_active":"2024-03-24T11:40:00.376313+0000","last_peered":"2024-03-24T11:40:00.376313+0000","last_clean":"2024-03-24T11:40:00.376313+0000","last_became_active":"2024-03-24T11:39:29.123199+0000","last_became_peered":"2024-03-24T11:39:29.123199+0000","last_unstale":"2024-03-24T11:40:00.376313+0000","last_undegraded":"2024-03-24T11:40:00.376313+0000","last_fullsized":"2024-03-24T11:40:00.376313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:38.696993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377962+0000","last_change":"2024-03-24T11:39:26.336886+0000","last_active":"2024-03-24T11:40:00.377962+0000","last_peered":"2024-03-24T11:40:00.377962+0000","last_clean":"2024-03-24T11:40:00.377962+0000","last_became_active":"2024-03-24T11:39:26.336469+0000","last_became_peered":"2024-03-24T11:39:26.336469+0000","last_unstale":"2024-03-24T11:40:00.377962+0000","last_undegraded":"2024-03-24T11:40:00.377962+0000","last_fullsized":"2024-03-24T11:40:00.377962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:11.100949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376521+0000","last_change":"2024-03-24T11:39:22.274217+0000","last_active":"2024-03-24T11:40:00.376521+0000","last_peered":"2024-03-24T11:40:00.376521+0000","last_clean":"2024-03-24T11:40:00.376521+0000","last_became_active":"2024-03-24T11:39:22.273533+0000","last_became_peered":"2024-03-24T11:39:22.273533+0000","last_unstale":"2024-03-24T11:40:00.376521+0000","last_undegraded":"2024-03-24T11:40:00.376521+0000","last_fullsized":"2024-03-24T11:40:00.376521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:42.837194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373537+0000","last_change":"2024-03-24T11:39:24.333633+0000","last_active":"2024-03-24T11:40:00.373537+0000","last_peered":"2024-03-24T11:40:00.373537+0000","last_clean":"2024-03-24T11:40:00.373537+0000","last_became_active":"2024-03-24T11:39:24.333314+0000","last_became_peered":"2024-03-24T11:39:24.333314+0000","last_unstale":"2024-03-24T11:40:00.373537+0000","last_undegraded":"2024-03-24T11:40:00.373537+0000","last_fullsized":"2024-03-24T11:40:00.373537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:41.046356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497963+0000","last_change":"2024-03-24T11:39:28.358462+0000","last_active":"2024-03-24T11:40:00.497963+0000","last_peered":"2024-03-24T11:40:00.497963+0000","last_clean":"2024-03-24T11:40:00.497963+0000","last_became_active":"2024-03-24T11:39:28.358184+0000","last_became_peered":"2024-03-24T11:39:28.358184+0000","last_unstale":"2024-03-24T11:40:00.497963+0000","last_undegraded":"2024-03-24T11:40:00.497963+0000","last_fullsized":"2024-03-24T11:40:00.497963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:37:05.786244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330555+0000","last_change":"2024-03-24T11:39:26.342708+0000","last_active":"2024-03-24T11:40:42.330555+0000","last_peered":"2024-03-24T11:40:42.330555+0000","last_clean":"2024-03-24T11:40:42.330555+0000","last_became_active":"2024-03-24T11:39:26.342382+0000","last_became_peered":"2024-03-24T11:39:26.342382+0000","last_unstale":"2024-03-24T11:40:42.330555+0000","last_undegraded":"2024-03-24T11:40:42.330555+0000","last_fullsized":"2024-03-24T11:40:42.330555+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:47:07.451790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214211+0000","last_change":"2024-03-24T11:39:22.276063+0000","last_active":"2024-03-24T11:40:01.214211+0000","last_peered":"2024-03-24T11:40:01.214211+0000","last_clean":"2024-03-24T11:40:01.214211+0000","last_became_active":"2024-03-24T11:39:22.275726+0000","last_became_peered":"2024-03-24T11:39:22.275726+0000","last_unstale":"2024-03-24T11:40:01.214211+0000","last_undegraded":"2024-03-24T11:40:01.214211+0000","last_fullsized":"2024-03-24T11:40:01.214211+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:05.420375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215358+0000","last_change":"2024-03-24T11:39:24.327617+0000","last_active":"2024-03-24T11:40:01.215358+0000","last_peered":"2024-03-24T11:40:01.215358+0000","last_clean":"2024-03-24T11:40:01.215358+0000","last_became_active":"2024-03-24T11:39:24.327186+0000","last_became_peered":"2024-03-24T11:39:24.327186+0000","last_unstale":"2024-03-24T11:40:01.215358+0000","last_undegraded":"2024-03-24T11:40:01.215358+0000","last_fullsized":"2024-03-24T11:40:01.215358+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:14.082702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215205+0000","last_change":"2024-03-24T11:39:28.350429+0000","last_active":"2024-03-24T11:40:01.215205+0000","last_peered":"2024-03-24T11:40:01.215205+0000","last_clean":"2024-03-24T11:40:01.215205+0000","last_became_active":"2024-03-24T11:39:28.350043+0000","last_became_peered":"2024-03-24T11:39:28.350043+0000","last_unstale":"2024-03-24T11:40:01.215205+0000","last_undegraded":"2024-03-24T11:40:01.215205+0000","last_fullsized":"2024-03-24T11:40:01.215205+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:26:47.934591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215565+0000","last_change":"2024-03-24T11:39:26.340610+0000","last_active":"2024-03-24T11:40:01.215565+0000","last_peered":"2024-03-24T11:40:01.215565+0000","last_clean":"2024-03-24T11:40:01.215565+0000","last_became_active":"2024-03-24T11:39:26.340262+0000","last_became_peered":"2024-03-24T11:39:26.340262+0000","last_unstale":"2024-03-24T11:40:01.215565+0000","last_undegraded":"2024-03-24T11:40:01.215565+0000","last_fullsized":"2024-03-24T11:40:01.215565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:02:04.790525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373359+0000","last_change":"2024-03-24T11:39:22.276032+0000","last_active":"2024-03-24T11:40:00.373359+0000","last_peered":"2024-03-24T11:40:00.373359+0000","last_clean":"2024-03-24T11:40:00.373359+0000","last_became_active":"2024-03-24T11:39:22.275629+0000","last_became_peered":"2024-03-24T11:39:22.275629+0000","last_unstale":"2024-03-24T11:40:00.373359+0000","last_undegraded":"2024-03-24T11:40:00.373359+0000","last_fullsized":"2024-03-24T11:40:00.373359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:10.405957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112961+0000","last_change":"2024-03-24T11:39:24.325862+0000","last_active":"2024-03-24T11:40:01.112961+0000","last_peered":"2024-03-24T11:40:01.112961+0000","last_clean":"2024-03-24T11:40:01.112961+0000","last_became_active":"2024-03-24T11:39:24.325536+0000","last_became_peered":"2024-03-24T11:39:24.325536+0000","last_unstale":"2024-03-24T11:40:01.112961+0000","last_undegraded":"2024-03-24T11:40:01.112961+0000","last_fullsized":"2024-03-24T11:40:01.112961+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:29.444437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497787+0000","last_change":"2024-03-24T11:39:28.355666+0000","last_active":"2024-03-24T11:40:00.497787+0000","last_peered":"2024-03-24T11:40:00.497787+0000","last_clean":"2024-03-24T11:40:00.497787+0000","last_became_active":"2024-03-24T11:39:28.355337+0000","last_became_peered":"2024-03-24T11:39:28.355337+0000","last_unstale":"2024-03-24T11:40:00.497787+0000","last_undegraded":"2024-03-24T11:40:00.497787+0000","last_fullsized":"2024-03-24T11:40:00.497787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:00.651233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112336+0000","last_change":"2024-03-24T11:39:26.344818+0000","last_active":"2024-03-24T11:40:01.112336+0000","last_peered":"2024-03-24T11:40:01.112336+0000","last_clean":"2024-03-24T11:40:01.112336+0000","last_became_active":"2024-03-24T11:39:26.344336+0000","last_became_peered":"2024-03-24T11:39:26.344336+0000","last_unstale":"2024-03-24T11:40:01.112336+0000","last_undegraded":"2024-03-24T11:40:01.112336+0000","last_fullsized":"2024-03-24T11:40:01.112336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:38:15.244685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215224+0000","last_change":"2024-03-24T11:39:22.279784+0000","last_active":"2024-03-24T11:40:01.215224+0000","last_peered":"2024-03-24T11:40:01.215224+0000","last_clean":"2024-03-24T11:40:01.215224+0000","last_became_active":"2024-03-24T11:39:22.279312+0000","last_became_peered":"2024-03-24T11:39:22.279312+0000","last_unstale":"2024-03-24T11:40:01.215224+0000","last_undegraded":"2024-03-24T11:40:01.215224+0000","last_fullsized":"2024-03-24T11:40:01.215224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:33:35.962134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373460+0000","last_change":"2024-03-24T11:39:24.331404+0000","last_active":"2024-03-24T11:40:00.373460+0000","last_peered":"2024-03-24T11:40:00.373460+0000","last_clean":"2024-03-24T11:40:00.373460+0000","last_became_active":"2024-03-24T11:39:24.330164+0000","last_became_peered":"2024-03-24T11:39:24.330164+0000","last_unstale":"2024-03-24T11:40:00.373460+0000","last_undegraded":"2024-03-24T11:40:00.373460+0000","last_fullsized":"2024-03-24T11:40:00.373460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:19:18.995381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373306+0000","last_change":"2024-03-24T11:39:29.121239+0000","last_active":"2024-03-24T11:40:00.373306+0000","last_peered":"2024-03-24T11:40:00.373306+0000","last_clean":"2024-03-24T11:40:00.373306+0000","last_became_active":"2024-03-24T11:39:29.120752+0000","last_became_peered":"2024-03-24T11:39:29.120752+0000","last_unstale":"2024-03-24T11:40:00.373306+0000","last_undegraded":"2024-03-24T11:40:00.373306+0000","last_fullsized":"2024-03-24T11:40:00.373306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:15:11.649187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498035+0000","last_change":"2024-03-24T11:39:26.347323+0000","last_active":"2024-03-24T11:40:00.498035+0000","last_peered":"2024-03-24T11:40:00.498035+0000","last_clean":"2024-03-24T11:40:00.498035+0000","last_became_active":"2024-03-24T11:39:26.347002+0000","last_became_peered":"2024-03-24T11:39:26.347002+0000","last_unstale":"2024-03-24T11:40:00.498035+0000","last_undegraded":"2024-03-24T11:40:00.498035+0000","last_fullsized":"2024-03-24T11:40:00.498035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:55.314835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482561+0000","last_change":"2024-03-24T11:39:22.276924+0000","last_active":"2024-03-24T11:40:00.482561+0000","last_peered":"2024-03-24T11:40:00.482561+0000","last_clean":"2024-03-24T11:40:00.482561+0000","last_became_active":"2024-03-24T11:39:22.276608+0000","last_became_peered":"2024-03-24T11:39:22.276608+0000","last_unstale":"2024-03-24T11:40:00.482561+0000","last_undegraded":"2024-03-24T11:40:00.482561+0000","last_fullsized":"2024-03-24T11:40:00.482561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:30.540917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375637+0000","last_change":"2024-03-24T11:39:24.327208+0000","last_active":"2024-03-24T11:40:00.375637+0000","last_peered":"2024-03-24T11:40:00.375637+0000","last_clean":"2024-03-24T11:40:00.375637+0000","last_became_active":"2024-03-24T11:39:24.326219+0000","last_became_peered":"2024-03-24T11:39:24.326219+0000","last_unstale":"2024-03-24T11:40:00.375637+0000","last_undegraded":"2024-03-24T11:40:00.375637+0000","last_fullsized":"2024-03-24T11:40:00.375637+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:08.619718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214004+0000","last_change":"2024-03-24T11:39:28.359665+0000","last_active":"2024-03-24T11:40:01.214004+0000","last_peered":"2024-03-24T11:40:01.214004+0000","last_clean":"2024-03-24T11:40:01.214004+0000","last_became_active":"2024-03-24T11:39:28.359279+0000","last_became_peered":"2024-03-24T11:39:28.359279+0000","last_unstale":"2024-03-24T11:40:01.214004+0000","last_undegraded":"2024-03-24T11:40:01.214004+0000","last_fullsized":"2024-03-24T11:40:01.214004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:04:25.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373588+0000","last_change":"2024-03-24T11:39:26.338454+0000","last_active":"2024-03-24T11:40:00.373588+0000","last_peered":"2024-03-24T11:40:00.373588+0000","last_clean":"2024-03-24T11:40:00.373588+0000","last_became_active":"2024-03-24T11:39:26.338125+0000","last_became_peered":"2024-03-24T11:39:26.338125+0000","last_unstale":"2024-03-24T11:40:00.373588+0000","last_undegraded":"2024-03-24T11:40:00.373588+0000","last_fullsized":"2024-03-24T11:40:00.373588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:47:18.844947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498200+0000","last_change":"2024-03-24T11:39:22.275813+0000","last_active":"2024-03-24T11:40:00.498200+0000","last_peered":"2024-03-24T11:40:00.498200+0000","last_clean":"2024-03-24T11:40:00.498200+0000","last_became_active":"2024-03-24T11:39:22.274712+0000","last_became_peered":"2024-03-24T11:39:22.274712+0000","last_unstale":"2024-03-24T11:40:00.498200+0000","last_undegraded":"2024-03-24T11:40:00.498200+0000","last_fullsized":"2024-03-24T11:40:00.498200+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:20:48.875472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215281+0000","last_change":"2024-03-24T11:39:24.328109+0000","last_active":"2024-03-24T11:40:01.215281+0000","last_peered":"2024-03-24T11:40:01.215281+0000","last_clean":"2024-03-24T11:40:01.215281+0000","last_became_active":"2024-03-24T11:39:24.327638+0000","last_became_peered":"2024-03-24T11:39:24.327638+0000","last_unstale":"2024-03-24T11:40:01.215281+0000","last_undegraded":"2024-03-24T11:40:01.215281+0000","last_fullsized":"2024-03-24T11:40:01.215281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:25:06.857407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375317+0000","last_change":"2024-03-24T11:39:28.361076+0000","last_active":"2024-03-24T11:40:00.375317+0000","last_peered":"2024-03-24T11:40:00.375317+0000","last_clean":"2024-03-24T11:40:00.375317+0000","last_became_active":"2024-03-24T11:39:28.360310+0000","last_became_peered":"2024-03-24T11:39:28.360310+0000","last_unstale":"2024-03-24T11:40:00.375317+0000","last_undegraded":"2024-03-24T11:40:00.375317+0000","last_fullsized":"2024-03-24T11:40:00.375317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:16:05.714030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214732+0000","last_change":"2024-03-24T11:39:26.341302+0000","last_active":"2024-03-24T11:40:01.214732+0000","last_peered":"2024-03-24T11:40:01.214732+0000","last_clean":"2024-03-24T11:40:01.214732+0000","last_became_active":"2024-03-24T11:39:26.340751+0000","last_became_peered":"2024-03-24T11:39:26.340751+0000","last_unstale":"2024-03-24T11:40:01.214732+0000","last_undegraded":"2024-03-24T11:40:01.214732+0000","last_fullsized":"2024-03-24T11:40:01.214732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:29.203219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378229+0000","last_change":"2024-03-24T11:39:22.269859+0000","last_active":"2024-03-24T11:40:00.378229+0000","last_peered":"2024-03-24T11:40:00.378229+0000","last_clean":"2024-03-24T11:40:00.378229+0000","last_became_active":"2024-03-24T11:39:22.269574+0000","last_became_peered":"2024-03-24T11:39:22.269574+0000","last_unstale":"2024-03-24T11:40:00.378229+0000","last_undegraded":"2024-03-24T11:40:00.378229+0000","last_fullsized":"2024-03-24T11:40:00.378229+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:37:03.492581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376234+0000","last_change":"2024-03-24T11:39:24.330079+0000","last_active":"2024-03-24T11:40:00.376234+0000","last_peered":"2024-03-24T11:40:00.376234+0000","last_clean":"2024-03-24T11:40:00.376234+0000","last_became_active":"2024-03-24T11:39:24.329703+0000","last_became_peered":"2024-03-24T11:39:24.329703+0000","last_unstale":"2024-03-24T11:40:00.376234+0000","last_undegraded":"2024-03-24T11:40:00.376234+0000","last_fullsized":"2024-03-24T11:40:00.376234+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:02.144542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215145+0000","last_change":"2024-03-24T11:39:28.361156+0000","last_active":"2024-03-24T11:40:01.215145+0000","last_peered":"2024-03-24T11:40:01.215145+0000","last_clean":"2024-03-24T11:40:01.215145+0000","last_became_active":"2024-03-24T11:39:28.360836+0000","last_became_peered":"2024-03-24T11:39:28.360836+0000","last_unstale":"2024-03-24T11:40:01.215145+0000","last_undegraded":"2024-03-24T11:40:01.215145+0000","last_fullsized":"2024-03-24T11:40:01.215145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:12.265272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330529+0000","last_change":"2024-03-24T11:39:26.341714+0000","last_active":"2024-03-24T11:40:42.330529+0000","last_peered":"2024-03-24T11:40:42.330529+0000","last_clean":"2024-03-24T11:40:42.330529+0000","last_became_active":"2024-03-24T11:39:26.341233+0000","last_became_peered":"2024-03-24T11:39:26.341233+0000","last_unstale":"2024-03-24T11:40:42.330529+0000","last_undegraded":"2024-03-24T11:40:42.330529+0000","last_fullsized":"2024-03-24T11:40:42.330529+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:52.851836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497855+0000","last_change":"2024-03-24T11:39:22.274274+0000","last_active":"2024-03-24T11:40:00.497855+0000","last_peered":"2024-03-24T11:40:00.497855+0000","last_clean":"2024-03-24T11:40:00.497855+0000","last_became_active":"2024-03-24T11:39:22.273733+0000","last_became_peered":"2024-03-24T11:39:22.273733+0000","last_unstale":"2024-03-24T11:40:00.497855+0000","last_undegraded":"2024-03-24T11:40:00.497855+0000","last_fullsized":"2024-03-24T11:40:00.497855+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:34:38.534882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373925+0000","last_change":"2024-03-24T11:39:24.331998+0000","last_active":"2024-03-24T11:40:00.373925+0000","last_peered":"2024-03-24T11:40:00.373925+0000","last_clean":"2024-03-24T11:40:00.373925+0000","last_became_active":"2024-03-24T11:39:24.331694+0000","last_became_peered":"2024-03-24T11:39:24.331694+0000","last_unstale":"2024-03-24T11:40:00.373925+0000","last_undegraded":"2024-03-24T11:40:00.373925+0000","last_fullsized":"2024-03-24T11:40:00.373925+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:24:52.556163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378470+0000","last_change":"2024-03-24T11:39:28.348376+0000","last_active":"2024-03-24T11:40:00.378470+0000","last_peered":"2024-03-24T11:40:00.378470+0000","last_clean":"2024-03-24T11:40:00.378470+0000","last_became_active":"2024-03-24T11:39:28.347911+0000","last_became_peered":"2024-03-24T11:39:28.347911+0000","last_unstale":"2024-03-24T11:40:00.378470+0000","last_undegraded":"2024-03-24T11:40:00.378470+0000","last_fullsized":"2024-03-24T11:40:00.378470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:59:46.740938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330697+0000","last_change":"2024-03-24T11:39:26.340225+0000","last_active":"2024-03-24T11:40:42.330697+0000","last_peered":"2024-03-24T11:40:42.330697+0000","last_clean":"2024-03-24T11:40:42.330697+0000","last_became_active":"2024-03-24T11:39:26.339897+0000","last_became_peered":"2024-03-24T11:39:26.339897+0000","last_unstale":"2024-03-24T11:40:42.330697+0000","last_undegraded":"2024-03-24T11:40:42.330697+0000","last_fullsized":"2024-03-24T11:40:42.330697+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:44:47.566066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214086+0000","last_change":"2024-03-24T11:39:22.278002+0000","last_active":"2024-03-24T11:40:01.214086+0000","last_peered":"2024-03-24T11:40:01.214086+0000","last_clean":"2024-03-24T11:40:01.214086+0000","last_became_active":"2024-03-24T11:39:22.277702+0000","last_became_peered":"2024-03-24T11:39:22.277702+0000","last_unstale":"2024-03-24T11:40:01.214086+0000","last_undegraded":"2024-03-24T11:40:01.214086+0000","last_fullsized":"2024-03-24T11:40:01.214086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:35:18.769871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482769+0000","last_change":"2024-03-24T11:39:24.324557+0000","last_active":"2024-03-24T11:40:00.482769+0000","last_peered":"2024-03-24T11:40:00.482769+0000","last_clean":"2024-03-24T11:40:00.482769+0000","last_became_active":"2024-03-24T11:39:24.324203+0000","last_became_peered":"2024-03-24T11:39:24.324203+0000","last_unstale":"2024-03-24T11:40:00.482769+0000","last_undegraded":"2024-03-24T11:40:00.482769+0000","last_fullsized":"2024-03-24T11:40:00.482769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:04:12.430126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373999+0000","last_change":"2024-03-24T11:39:28.357168+0000","last_active":"2024-03-24T11:40:00.373999+0000","last_peered":"2024-03-24T11:40:00.373999+0000","last_clean":"2024-03-24T11:40:00.373999+0000","last_became_active":"2024-03-24T11:39:28.356800+0000","last_became_peered":"2024-03-24T11:39:28.356800+0000","last_unstale":"2024-03-24T11:40:00.373999+0000","last_undegraded":"2024-03-24T11:40:00.373999+0000","last_fullsized":"2024-03-24T11:40:00.373999+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:43:55.200971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376231+0000","last_change":"2024-03-24T11:39:26.339993+0000","last_active":"2024-03-24T11:40:00.376231+0000","last_peered":"2024-03-24T11:40:00.376231+0000","last_clean":"2024-03-24T11:40:00.376231+0000","last_became_active":"2024-03-24T11:39:26.339644+0000","last_became_peered":"2024-03-24T11:39:26.339644+0000","last_unstale":"2024-03-24T11:40:00.376231+0000","last_undegraded":"2024-03-24T11:40:00.376231+0000","last_fullsized":"2024-03-24T11:40:00.376231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:00:14.664377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497813+0000","last_change":"2024-03-24T11:39:22.274140+0000","last_active":"2024-03-24T11:40:00.497813+0000","last_peered":"2024-03-24T11:40:00.497813+0000","last_clean":"2024-03-24T11:40:00.497813+0000","last_became_active":"2024-03-24T11:39:22.273798+0000","last_became_peered":"2024-03-24T11:39:22.273798+0000","last_unstale":"2024-03-24T11:40:00.497813+0000","last_undegraded":"2024-03-24T11:40:00.497813+0000","last_fullsized":"2024-03-24T11:40:00.497813+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:14:22.212219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215442+0000","last_change":"2024-03-24T11:39:24.324454+0000","last_active":"2024-03-24T11:40:01.215442+0000","last_peered":"2024-03-24T11:40:01.215442+0000","last_clean":"2024-03-24T11:40:01.215442+0000","last_became_active":"2024-03-24T11:39:24.324115+0000","last_became_peered":"2024-03-24T11:39:24.324115+0000","last_unstale":"2024-03-24T11:40:01.215442+0000","last_undegraded":"2024-03-24T11:40:01.215442+0000","last_fullsized":"2024-03-24T11:40:01.215442+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:37:32.036427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498166+0000","last_change":"2024-03-24T11:39:29.121414+0000","last_active":"2024-03-24T11:40:00.498166+0000","last_peered":"2024-03-24T11:40:00.498166+0000","last_clean":"2024-03-24T11:40:00.498166+0000","last_became_active":"2024-03-24T11:39:29.121062+0000","last_became_peered":"2024-03-24T11:39:29.121062+0000","last_unstale":"2024-03-24T11:40:00.498166+0000","last_undegraded":"2024-03-24T11:40:00.498166+0000","last_fullsized":"2024-03-24T11:40:00.498166+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:13.679597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482607+0000","last_change":"2024-03-24T11:39:26.339373+0000","last_active":"2024-03-24T11:40:00.482607+0000","last_peered":"2024-03-24T11:40:00.482607+0000","last_clean":"2024-03-24T11:40:00.482607+0000","last_became_active":"2024-03-24T11:39:26.338904+0000","last_became_peered":"2024-03-24T11:39:26.338904+0000","last_unstale":"2024-03-24T11:40:00.482607+0000","last_undegraded":"2024-03-24T11:40:00.482607+0000","last_fullsized":"2024-03-24T11:40:00.482607+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:46:47.552299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112318+0000","last_change":"2024-03-24T11:39:22.274494+0000","last_active":"2024-03-24T11:40:01.112318+0000","last_peered":"2024-03-24T11:40:01.112318+0000","last_clean":"2024-03-24T11:40:01.112318+0000","last_became_active":"2024-03-24T11:39:22.274217+0000","last_became_peered":"2024-03-24T11:39:22.274217+0000","last_unstale":"2024-03-24T11:40:01.112318+0000","last_undegraded":"2024-03-24T11:40:01.112318+0000","last_fullsized":"2024-03-24T11:40:01.112318+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:33:12.203983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376399+0000","last_change":"2024-03-24T11:39:24.333126+0000","last_active":"2024-03-24T11:40:00.376399+0000","last_peered":"2024-03-24T11:40:00.376399+0000","last_clean":"2024-03-24T11:40:00.376399+0000","last_became_active":"2024-03-24T11:39:24.332828+0000","last_became_peered":"2024-03-24T11:39:24.332828+0000","last_unstale":"2024-03-24T11:40:00.376399+0000","last_undegraded":"2024-03-24T11:40:00.376399+0000","last_fullsized":"2024-03-24T11:40:00.376399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:30:49.938650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482323+0000","last_change":"2024-03-24T11:39:29.121002+0000","last_active":"2024-03-24T11:40:00.482323+0000","last_peered":"2024-03-24T11:40:00.482323+0000","last_clean":"2024-03-24T11:40:00.482323+0000","last_became_active":"2024-03-24T11:39:29.120720+0000","last_became_peered":"2024-03-24T11:39:29.120720+0000","last_unstale":"2024-03-24T11:40:00.482323+0000","last_undegraded":"2024-03-24T11:40:00.482323+0000","last_fullsized":"2024-03-24T11:40:00.482323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:28:20.422181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498419+0000","last_change":"2024-03-24T11:39:22.279264+0000","last_active":"2024-03-24T11:40:00.498419+0000","last_peered":"2024-03-24T11:40:00.498419+0000","last_clean":"2024-03-24T11:40:00.498419+0000","last_became_active":"2024-03-24T11:39:22.278908+0000","last_became_peered":"2024-03-24T11:39:22.278908+0000","last_unstale":"2024-03-24T11:40:00.498419+0000","last_undegraded":"2024-03-24T11:40:00.498419+0000","last_fullsized":"2024-03-24T11:40:00.498419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:52:18.918085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482092+0000","last_change":"2024-03-24T11:39:24.318820+0000","last_active":"2024-03-24T11:40:00.482092+0000","last_peered":"2024-03-24T11:40:00.482092+0000","last_clean":"2024-03-24T11:40:00.482092+0000","last_became_active":"2024-03-24T11:39:24.318488+0000","last_became_peered":"2024-03-24T11:39:24.318488+0000","last_unstale":"2024-03-24T11:40:00.482092+0000","last_undegraded":"2024-03-24T11:40:00.482092+0000","last_fullsized":"2024-03-24T11:40:00.482092+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:50:11.994317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330790+0000","last_change":"2024-03-24T11:39:26.342727+0000","last_active":"2024-03-24T11:40:42.330790+0000","last_peered":"2024-03-24T11:40:42.330790+0000","last_clean":"2024-03-24T11:40:42.330790+0000","last_became_active":"2024-03-24T11:39:26.342201+0000","last_became_peered":"2024-03-24T11:39:26.342201+0000","last_unstale":"2024-03-24T11:40:42.330790+0000","last_undegraded":"2024-03-24T11:40:42.330790+0000","last_fullsized":"2024-03-24T11:40:42.330790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:38.176799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373407+0000","last_change":"2024-03-24T11:39:28.357336+0000","last_active":"2024-03-24T11:40:00.373407+0000","last_peered":"2024-03-24T11:40:00.373407+0000","last_clean":"2024-03-24T11:40:00.373407+0000","last_became_active":"2024-03-24T11:39:28.357024+0000","last_became_peered":"2024-03-24T11:39:28.357024+0000","last_unstale":"2024-03-24T11:40:00.373407+0000","last_undegraded":"2024-03-24T11:40:00.373407+0000","last_fullsized":"2024-03-24T11:40:00.373407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:20.869919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498352+0000","last_change":"2024-03-24T11:39:22.279163+0000","last_active":"2024-03-24T11:40:00.498352+0000","last_peered":"2024-03-24T11:40:00.498352+0000","last_clean":"2024-03-24T11:40:00.498352+0000","last_became_active":"2024-03-24T11:39:22.278855+0000","last_became_peered":"2024-03-24T11:39:22.278855+0000","last_unstale":"2024-03-24T11:40:00.498352+0000","last_undegraded":"2024-03-24T11:40:00.498352+0000","last_fullsized":"2024-03-24T11:40:00.498352+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:09.826860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377031+0000","last_change":"2024-03-24T11:39:24.329936+0000","last_active":"2024-03-24T11:40:00.377031+0000","last_peered":"2024-03-24T11:40:00.377031+0000","last_clean":"2024-03-24T11:40:00.377031+0000","last_became_active":"2024-03-24T11:39:24.329578+0000","last_became_peered":"2024-03-24T11:39:24.329578+0000","last_unstale":"2024-03-24T11:40:00.377031+0000","last_undegraded":"2024-03-24T11:40:00.377031+0000","last_fullsized":"2024-03-24T11:40:00.377031+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:17:51.128016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373035+0000","last_change":"2024-03-24T11:39:26.338031+0000","last_active":"2024-03-24T11:40:00.373035+0000","last_peered":"2024-03-24T11:40:00.373035+0000","last_clean":"2024-03-24T11:40:00.373035+0000","last_became_active":"2024-03-24T11:39:26.337661+0000","last_became_peered":"2024-03-24T11:39:26.337661+0000","last_unstale":"2024-03-24T11:40:00.373035+0000","last_undegraded":"2024-03-24T11:40:00.373035+0000","last_fullsized":"2024-03-24T11:40:00.373035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:03.905497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497650+0000","last_change":"2024-03-24T11:39:28.355496+0000","last_active":"2024-03-24T11:40:00.497650+0000","last_peered":"2024-03-24T11:40:00.497650+0000","last_clean":"2024-03-24T11:40:00.497650+0000","last_became_active":"2024-03-24T11:39:28.355197+0000","last_became_peered":"2024-03-24T11:39:28.355197+0000","last_unstale":"2024-03-24T11:40:00.497650+0000","last_undegraded":"2024-03-24T11:40:00.497650+0000","last_fullsized":"2024-03-24T11:40:00.497650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:42.142473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373386+0000","last_change":"2024-03-24T11:39:26.339649+0000","last_active":"2024-03-24T11:40:00.373386+0000","last_peered":"2024-03-24T11:40:00.373386+0000","last_clean":"2024-03-24T11:40:00.373386+0000","last_became_active":"2024-03-24T11:39:26.339188+0000","last_became_peered":"2024-03-24T11:39:26.339188+0000","last_unstale":"2024-03-24T11:40:00.373386+0000","last_undegraded":"2024-03-24T11:40:00.373386+0000","last_fullsized":"2024-03-24T11:40:00.373386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:50:18.916718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498307+0000","last_change":"2024-03-24T11:39:22.272264+0000","last_active":"2024-03-24T11:40:00.498307+0000","last_peered":"2024-03-24T11:40:00.498307+0000","last_clean":"2024-03-24T11:40:00.498307+0000","last_became_active":"2024-03-24T11:39:22.271945+0000","last_became_peered":"2024-03-24T11:39:22.271945+0000","last_unstale":"2024-03-24T11:40:00.498307+0000","last_undegraded":"2024-03-24T11:40:00.498307+0000","last_fullsized":"2024-03-24T11:40:00.498307+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:50.400313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112758+0000","last_change":"2024-03-24T11:39:24.326143+0000","last_active":"2024-03-24T11:40:01.112758+0000","last_peered":"2024-03-24T11:40:01.112758+0000","last_clean":"2024-03-24T11:40:01.112758+0000","last_became_active":"2024-03-24T11:39:24.325479+0000","last_became_peered":"2024-03-24T11:39:24.325479+0000","last_unstale":"2024-03-24T11:40:01.112758+0000","last_undegraded":"2024-03-24T11:40:01.112758+0000","last_fullsized":"2024-03-24T11:40:01.112758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:02:58.851736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214026+0000","last_change":"2024-03-24T11:39:28.357405+0000","last_active":"2024-03-24T11:40:01.214026+0000","last_peered":"2024-03-24T11:40:01.214026+0000","last_clean":"2024-03-24T11:40:01.214026+0000","last_became_active":"2024-03-24T11:39:28.357093+0000","last_became_peered":"2024-03-24T11:39:28.357093+0000","last_unstale":"2024-03-24T11:40:01.214026+0000","last_undegraded":"2024-03-24T11:40:01.214026+0000","last_fullsized":"2024-03-24T11:40:01.214026+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:35:43.876501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373177+0000","last_change":"2024-03-24T11:39:26.333304+0000","last_active":"2024-03-24T11:40:00.373177+0000","last_peered":"2024-03-24T11:40:00.373177+0000","last_clean":"2024-03-24T11:40:00.373177+0000","last_became_active":"2024-03-24T11:39:26.332971+0000","last_became_peered":"2024-03-24T11:39:26.332971+0000","last_unstale":"2024-03-24T11:40:00.373177+0000","last_undegraded":"2024-03-24T11:40:00.373177+0000","last_fullsized":"2024-03-24T11:40:00.373177+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:39:11.165619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482417+0000","last_change":"2024-03-24T11:39:22.270901+0000","last_active":"2024-03-24T11:40:00.482417+0000","last_peered":"2024-03-24T11:40:00.482417+0000","last_clean":"2024-03-24T11:40:00.482417+0000","last_became_active":"2024-03-24T11:39:22.270402+0000","last_became_peered":"2024-03-24T11:39:22.270402+0000","last_unstale":"2024-03-24T11:40:00.482417+0000","last_undegraded":"2024-03-24T11:40:00.482417+0000","last_fullsized":"2024-03-24T11:40:00.482417+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:59:17.729541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215800+0000","last_change":"2024-03-24T11:39:24.329920+0000","last_active":"2024-03-24T11:40:01.215800+0000","last_peered":"2024-03-24T11:40:01.215800+0000","last_clean":"2024-03-24T11:40:01.215800+0000","last_became_active":"2024-03-24T11:39:24.329567+0000","last_became_peered":"2024-03-24T11:39:24.329567+0000","last_unstale":"2024-03-24T11:40:01.215800+0000","last_undegraded":"2024-03-24T11:40:01.215800+0000","last_fullsized":"2024-03-24T11:40:01.215800+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:11:52.558860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374284+0000","last_change":"2024-03-24T11:39:28.358245+0000","last_active":"2024-03-24T11:40:00.374284+0000","last_peered":"2024-03-24T11:40:00.374284+0000","last_clean":"2024-03-24T11:40:00.374284+0000","last_became_active":"2024-03-24T11:39:28.357926+0000","last_became_peered":"2024-03-24T11:39:28.357926+0000","last_unstale":"2024-03-24T11:40:00.374284+0000","last_undegraded":"2024-03-24T11:40:00.374284+0000","last_fullsized":"2024-03-24T11:40:00.374284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:52:35.025907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482136+0000","last_change":"2024-03-24T11:39:26.338093+0000","last_active":"2024-03-24T11:40:00.482136+0000","last_peered":"2024-03-24T11:40:00.482136+0000","last_clean":"2024-03-24T11:40:00.482136+0000","last_became_active":"2024-03-24T11:39:22.276126+0000","last_became_peered":"2024-03-24T11:39:22.276126+0000","last_unstale":"2024-03-24T11:40:00.482136+0000","last_undegraded":"2024-03-24T11:40:00.482136+0000","last_fullsized":"2024-03-24T11:40:00.482136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:08.764768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050133859999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376017+0000","last_change":"2024-03-24T11:39:22.278162+0000","last_active":"2024-03-24T11:40:00.376017+0000","last_peered":"2024-03-24T11:40:00.376017+0000","last_clean":"2024-03-24T11:40:00.376017+0000","last_became_active":"2024-03-24T11:39:22.274727+0000","last_became_peered":"2024-03-24T11:39:22.274727+0000","last_unstale":"2024-03-24T11:40:00.376017+0000","last_undegraded":"2024-03-24T11:40:00.376017+0000","last_fullsized":"2024-03-24T11:40:00.376017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:15:18.230675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":95,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330366+0000","last_change":"2024-03-24T11:39:24.334604+0000","last_active":"2024-03-24T11:40:42.330366+0000","last_peered":"2024-03-24T11:40:42.330366+0000","last_clean":"2024-03-24T11:40:42.330366+0000","last_became_active":"2024-03-24T11:39:24.334273+0000","last_became_peered":"2024-03-24T11:39:24.334273+0000","last_unstale":"2024-03-24T11:40:42.330366+0000","last_undegraded":"2024-03-24T11:40:42.330366+0000","last_fullsized":"2024-03-24T11:40:42.330366+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:27.778605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214808+0000","last_change":"2024-03-24T11:39:26.343791+0000","last_active":"2024-03-24T11:40:01.214808+0000","last_peered":"2024-03-24T11:40:01.214808+0000","last_clean":"2024-03-24T11:40:01.214808+0000","last_became_active":"2024-03-24T11:39:26.342830+0000","last_became_peered":"2024-03-24T11:39:26.342830+0000","last_unstale":"2024-03-24T11:40:01.214808+0000","last_undegraded":"2024-03-24T11:40:01.214808+0000","last_fullsized":"2024-03-24T11:40:01.214808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:55.962463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498399+0000","last_change":"2024-03-24T11:39:29.121058+0000","last_active":"2024-03-24T11:40:00.498399+0000","last_peered":"2024-03-24T11:40:00.498399+0000","last_clean":"2024-03-24T11:40:00.498399+0000","last_became_active":"2024-03-24T11:39:29.120626+0000","last_became_peered":"2024-03-24T11:39:29.120626+0000","last_unstale":"2024-03-24T11:40:00.498399+0000","last_undegraded":"2024-03-24T11:40:00.498399+0000","last_fullsized":"2024-03-24T11:40:00.498399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:52:46.205011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214627+0000","last_change":"2024-03-24T11:39:22.271768+0000","last_active":"2024-03-24T11:40:01.214627+0000","last_peered":"2024-03-24T11:40:01.214627+0000","last_clean":"2024-03-24T11:40:01.214627+0000","last_became_active":"2024-03-24T11:39:22.271342+0000","last_became_peered":"2024-03-24T11:39:22.271342+0000","last_unstale":"2024-03-24T11:40:01.214627+0000","last_undegraded":"2024-03-24T11:40:01.214627+0000","last_fullsized":"2024-03-24T11:40:01.214627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:01:03.354295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373079+0000","last_change":"2024-03-24T11:39:26.331241+0000","last_active":"2024-03-24T11:40:00.373079+0000","last_peered":"2024-03-24T11:40:00.373079+0000","last_clean":"2024-03-24T11:40:00.373079+0000","last_became_active":"2024-03-24T11:39:22.270767+0000","last_became_peered":"2024-03-24T11:39:22.270767+0000","last_unstale":"2024-03-24T11:40:00.373079+0000","last_undegraded":"2024-03-24T11:40:00.373079+0000","last_fullsized":"2024-03-24T11:40:00.373079+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:21:10.787895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051812070000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375105+0000","last_change":"2024-03-24T11:39:48.132616+0000","last_active":"2024-03-24T11:40:00.375105+0000","last_peered":"2024-03-24T11:40:00.375105+0000","last_clean":"2024-03-24T11:40:00.375105+0000","last_became_active":"2024-03-24T11:39:48.125273+0000","last_became_peered":"2024-03-24T11:39:48.125273+0000","last_unstale":"2024-03-24T11:40:00.375105+0000","last_undegraded":"2024-03-24T11:40:00.375105+0000","last_fullsized":"2024-03-24T11:40:00.375105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:52.137684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374451+0000","last_change":"2024-03-24T11:39:26.343946+0000","last_active":"2024-03-24T11:40:00.374451+0000","last_peered":"2024-03-24T11:40:00.374451+0000","last_clean":"2024-03-24T11:40:00.374451+0000","last_became_active":"2024-03-24T11:39:26.343594+0000","last_became_peered":"2024-03-24T11:39:26.343594+0000","last_unstale":"2024-03-24T11:40:00.374451+0000","last_undegraded":"2024-03-24T11:40:00.374451+0000","last_fullsized":"2024-03-24T11:40:00.374451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:11:53.949036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482610+0000","last_change":"2024-03-24T11:39:28.352754+0000","last_active":"2024-03-24T11:40:00.482610+0000","last_peered":"2024-03-24T11:40:00.482610+0000","last_clean":"2024-03-24T11:40:00.482610+0000","last_became_active":"2024-03-24T11:39:28.352205+0000","last_became_peered":"2024-03-24T11:39:28.352205+0000","last_unstale":"2024-03-24T11:40:00.482610+0000","last_undegraded":"2024-03-24T11:40:00.482610+0000","last_fullsized":"2024-03-24T11:40:00.482610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:13.096353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":276,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:02.481419+0000","last_change":"2024-03-24T11:39:12.747798+0000","last_active":"2024-03-24T11:40:02.481419+0000","last_peered":"2024-03-24T11:40:02.481419+0000","last_clean":"2024-03-24T11:40:02.481419+0000","last_became_active":"2024-03-24T11:39:12.747375+0000","last_became_peered":"2024-03-24T11:39:12.747375+0000","last_unstale":"2024-03-24T11:40:02.481419+0000","last_undegraded":"2024-03-24T11:40:02.481419+0000","last_fullsized":"2024-03-24T11:40:02.481419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:03.562175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:37:03.562175+0000","last_clean_scrub_stamp":"2024-03-24T11:37:03.562175+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:30:00.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376598+0000","last_change":"2024-03-24T11:39:22.272135+0000","last_active":"2024-03-24T11:40:00.376598+0000","last_peered":"2024-03-24T11:40:00.376598+0000","last_clean":"2024-03-24T11:40:00.376598+0000","last_became_active":"2024-03-24T11:39:22.271381+0000","last_became_peered":"2024-03-24T11:39:22.271381+0000","last_unstale":"2024-03-24T11:40:00.376598+0000","last_undegraded":"2024-03-24T11:40:00.376598+0000","last_fullsized":"2024-03-24T11:40:00.376598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:24.922413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":69,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330536+0000","last_change":"2024-03-24T11:39:24.324291+0000","last_active":"2024-03-24T11:40:42.330536+0000","last_peered":"2024-03-24T11:40:42.330536+0000","last_clean":"2024-03-24T11:40:42.330536+0000","last_became_active":"2024-03-24T11:39:24.323949+0000","last_became_peered":"2024-03-24T11:39:24.323949+0000","last_unstale":"2024-03-24T11:40:42.330536+0000","last_undegraded":"2024-03-24T11:40:42.330536+0000","last_fullsized":"2024-03-24T11:40:42.330536+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:47:59.001269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498143+0000","last_change":"2024-03-24T11:39:26.347455+0000","last_active":"2024-03-24T11:40:00.498143+0000","last_peered":"2024-03-24T11:40:00.498143+0000","last_clean":"2024-03-24T11:40:00.498143+0000","last_became_active":"2024-03-24T11:39:26.346834+0000","last_became_peered":"2024-03-24T11:39:26.346834+0000","last_unstale":"2024-03-24T11:40:00.498143+0000","last_undegraded":"2024-03-24T11:40:00.498143+0000","last_fullsized":"2024-03-24T11:40:00.498143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:41:15.692636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214415+0000","last_change":"2024-03-24T11:39:28.360890+0000","last_active":"2024-03-24T11:40:01.214415+0000","last_peered":"2024-03-24T11:40:01.214415+0000","last_clean":"2024-03-24T11:40:01.214415+0000","last_became_active":"2024-03-24T11:39:28.360589+0000","last_became_peered":"2024-03-24T11:39:28.360589+0000","last_unstale":"2024-03-24T11:40:01.214415+0000","last_undegraded":"2024-03-24T11:40:01.214415+0000","last_fullsized":"2024-03-24T11:40:01.214415+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:21.105809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214247+0000","last_change":"2024-03-24T11:39:22.278954+0000","last_active":"2024-03-24T11:40:01.214247+0000","last_peered":"2024-03-24T11:40:01.214247+0000","last_clean":"2024-03-24T11:40:01.214247+0000","last_became_active":"2024-03-24T11:39:22.278635+0000","last_became_peered":"2024-03-24T11:39:22.278635+0000","last_unstale":"2024-03-24T11:40:01.214247+0000","last_undegraded":"2024-03-24T11:40:01.214247+0000","last_fullsized":"2024-03-24T11:40:01.214247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:43:49.065824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":126,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:49.681337+0000","last_change":"2024-03-24T11:39:26.333995+0000","last_active":"2024-03-24T11:40:49.681337+0000","last_peered":"2024-03-24T11:40:49.681337+0000","last_clean":"2024-03-24T11:40:49.681337+0000","last_became_active":"2024-03-24T11:39:22.275391+0000","last_became_peered":"2024-03-24T11:39:22.275391+0000","last_unstale":"2024-03-24T11:40:49.681337+0000","last_undegraded":"2024-03-24T11:40:49.681337+0000","last_fullsized":"2024-03-24T11:40:49.681337+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:12:29.494066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012529869999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215236+0000","last_change":"2024-03-24T11:39:24.330195+0000","last_active":"2024-03-24T11:40:01.215236+0000","last_peered":"2024-03-24T11:40:01.215236+0000","last_clean":"2024-03-24T11:40:01.215236+0000","last_became_active":"2024-03-24T11:39:24.329552+0000","last_became_peered":"2024-03-24T11:39:24.329552+0000","last_unstale":"2024-03-24T11:40:01.215236+0000","last_undegraded":"2024-03-24T11:40:01.215236+0000","last_fullsized":"2024-03-24T11:40:01.215236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:57.235595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482844+0000","last_change":"2024-03-24T11:39:26.345010+0000","last_active":"2024-03-24T11:40:00.482844+0000","last_peered":"2024-03-24T11:40:00.482844+0000","last_clean":"2024-03-24T11:40:00.482844+0000","last_became_active":"2024-03-24T11:39:26.344464+0000","last_became_peered":"2024-03-24T11:39:26.344464+0000","last_unstale":"2024-03-24T11:40:00.482844+0000","last_undegraded":"2024-03-24T11:40:00.482844+0000","last_fullsized":"2024-03-24T11:40:00.482844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:36:38.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.213871+0000","last_change":"2024-03-24T11:39:29.121114+0000","last_active":"2024-03-24T11:40:01.213871+0000","last_peered":"2024-03-24T11:40:01.213871+0000","last_clean":"2024-03-24T11:40:01.213871+0000","last_became_active":"2024-03-24T11:39:29.120705+0000","last_became_peered":"2024-03-24T11:39:29.120705+0000","last_unstale":"2024-03-24T11:40:01.213871+0000","last_undegraded":"2024-03-24T11:40:01.213871+0000","last_fullsized":"2024-03-24T11:40:01.213871+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:34.237225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214646+0000","last_change":"2024-03-24T11:39:22.274676+0000","last_active":"2024-03-24T11:40:01.214646+0000","last_peered":"2024-03-24T11:40:01.214646+0000","last_clean":"2024-03-24T11:40:01.214646+0000","last_became_active":"2024-03-24T11:39:22.273941+0000","last_became_peered":"2024-03-24T11:39:22.273941+0000","last_unstale":"2024-03-24T11:40:01.214646+0000","last_undegraded":"2024-03-24T11:40:01.214646+0000","last_fullsized":"2024-03-24T11:40:01.214646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:33:53.159995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.216164+0000","last_change":"2024-03-24T11:39:24.323375+0000","last_active":"2024-03-24T11:40:01.216164+0000","last_peered":"2024-03-24T11:40:01.216164+0000","last_clean":"2024-03-24T11:40:01.216164+0000","last_became_active":"2024-03-24T11:39:24.323000+0000","last_became_peered":"2024-03-24T11:39:24.323000+0000","last_unstale":"2024-03-24T11:40:01.216164+0000","last_undegraded":"2024-03-24T11:40:01.216164+0000","last_fullsized":"2024-03-24T11:40:01.216164+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:37:42.703654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112774+0000","last_change":"2024-03-24T11:39:26.342326+0000","last_active":"2024-03-24T11:40:01.112774+0000","last_peered":"2024-03-24T11:40:01.112774+0000","last_clean":"2024-03-24T11:40:01.112774+0000","last_became_active":"2024-03-24T11:39:26.341988+0000","last_became_peered":"2024-03-24T11:39:26.341988+0000","last_unstale":"2024-03-24T11:40:01.112774+0000","last_undegraded":"2024-03-24T11:40:01.112774+0000","last_fullsized":"2024-03-24T11:40:01.112774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:40:21.773415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214110+0000","last_change":"2024-03-24T11:39:28.361252+0000","last_active":"2024-03-24T11:40:01.214110+0000","last_peered":"2024-03-24T11:40:01.214110+0000","last_clean":"2024-03-24T11:40:01.214110+0000","last_became_active":"2024-03-24T11:39:28.360947+0000","last_became_peered":"2024-03-24T11:39:28.360947+0000","last_unstale":"2024-03-24T11:40:01.214110+0000","last_undegraded":"2024-03-24T11:40:01.214110+0000","last_fullsized":"2024-03-24T11:40:01.214110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:00:52.933465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482676+0000","last_change":"2024-03-24T11:39:22.273361+0000","last_active":"2024-03-24T11:40:00.482676+0000","last_peered":"2024-03-24T11:40:00.482676+0000","last_clean":"2024-03-24T11:40:00.482676+0000","last_became_active":"2024-03-24T11:39:22.271889+0000","last_became_peered":"2024-03-24T11:39:22.271889+0000","last_unstale":"2024-03-24T11:40:00.482676+0000","last_undegraded":"2024-03-24T11:40:00.482676+0000","last_fullsized":"2024-03-24T11:40:00.482676+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:48.339910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215143+0000","last_change":"2024-03-24T11:39:24.330329+0000","last_active":"2024-03-24T11:40:01.215143+0000","last_peered":"2024-03-24T11:40:01.215143+0000","last_clean":"2024-03-24T11:40:01.215143+0000","last_became_active":"2024-03-24T11:39:24.329967+0000","last_became_peered":"2024-03-24T11:39:24.329967+0000","last_unstale":"2024-03-24T11:40:01.215143+0000","last_undegraded":"2024-03-24T11:40:01.215143+0000","last_fullsized":"2024-03-24T11:40:01.215143+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:09:05.137006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330363+0000","last_change":"2024-03-24T11:39:26.343215+0000","last_active":"2024-03-24T11:40:42.330363+0000","last_peered":"2024-03-24T11:40:42.330363+0000","last_clean":"2024-03-24T11:40:42.330363+0000","last_became_active":"2024-03-24T11:39:26.342926+0000","last_became_peered":"2024-03-24T11:39:26.342926+0000","last_unstale":"2024-03-24T11:40:42.330363+0000","last_undegraded":"2024-03-24T11:40:42.330363+0000","last_fullsized":"2024-03-24T11:40:42.330363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:27.233167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498161+0000","last_change":"2024-03-24T11:39:29.121026+0000","last_active":"2024-03-24T11:40:00.498161+0000","last_peered":"2024-03-24T11:40:00.498161+0000","last_clean":"2024-03-24T11:40:00.498161+0000","last_became_active":"2024-03-24T11:39:29.120687+0000","last_became_peered":"2024-03-24T11:39:29.120687+0000","last_unstale":"2024-03-24T11:40:00.498161+0000","last_undegraded":"2024-03-24T11:40:00.498161+0000","last_fullsized":"2024-03-24T11:40:00.498161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:43.442419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'2","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215334+0000","last_change":"2024-03-24T11:39:22.274126+0000","last_active":"2024-03-24T11:40:01.215334+0000","last_peered":"2024-03-24T11:40:01.215334+0000","last_clean":"2024-03-24T11:40:01.215334+0000","last_became_active":"2024-03-24T11:39:22.273797+0000","last_became_peered":"2024-03-24T11:39:22.273797+0000","last_unstale":"2024-03-24T11:40:01.215334+0000","last_undegraded":"2024-03-24T11:40:01.215334+0000","last_fullsized":"2024-03-24T11:40:01.215334+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:03:49.642051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376801+0000","last_change":"2024-03-24T11:39:24.321604+0000","last_active":"2024-03-24T11:40:00.376801+0000","last_peered":"2024-03-24T11:40:00.376801+0000","last_clean":"2024-03-24T11:40:00.376801+0000","last_became_active":"2024-03-24T11:39:24.321287+0000","last_became_peered":"2024-03-24T11:39:24.321287+0000","last_unstale":"2024-03-24T11:40:00.376801+0000","last_undegraded":"2024-03-24T11:40:00.376801+0000","last_fullsized":"2024-03-24T11:40:00.376801+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:39.522487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330437+0000","last_change":"2024-03-24T11:39:26.336388+0000","last_active":"2024-03-24T11:40:42.330437+0000","last_peered":"2024-03-24T11:40:42.330437+0000","last_clean":"2024-03-24T11:40:42.330437+0000","last_became_active":"2024-03-24T11:39:26.336079+0000","last_became_peered":"2024-03-24T11:39:26.336079+0000","last_unstale":"2024-03-24T11:40:42.330437+0000","last_undegraded":"2024-03-24T11:40:42.330437+0000","last_fullsized":"2024-03-24T11:40:42.330437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:52:21.640866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376273+0000","last_change":"2024-03-24T11:39:28.353587+0000","last_active":"2024-03-24T11:40:00.376273+0000","last_peered":"2024-03-24T11:40:00.376273+0000","last_clean":"2024-03-24T11:40:00.376273+0000","last_became_active":"2024-03-24T11:39:28.353261+0000","last_became_peered":"2024-03-24T11:39:28.353261+0000","last_unstale":"2024-03-24T11:40:00.376273+0000","last_undegraded":"2024-03-24T11:40:00.376273+0000","last_fullsized":"2024-03-24T11:40:00.376273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:20:30.018449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376033+0000","last_change":"2024-03-24T11:39:22.272865+0000","last_active":"2024-03-24T11:40:00.376033+0000","last_peered":"2024-03-24T11:40:00.376033+0000","last_clean":"2024-03-24T11:40:00.376033+0000","last_became_active":"2024-03-24T11:39:22.272573+0000","last_became_peered":"2024-03-24T11:39:22.272573+0000","last_unstale":"2024-03-24T11:40:00.376033+0000","last_undegraded":"2024-03-24T11:40:00.376033+0000","last_fullsized":"2024-03-24T11:40:00.376033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:37.543022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374687+0000","last_change":"2024-03-24T11:39:24.323346+0000","last_active":"2024-03-24T11:40:00.374687+0000","last_peered":"2024-03-24T11:40:00.374687+0000","last_clean":"2024-03-24T11:40:00.374687+0000","last_became_active":"2024-03-24T11:39:24.323056+0000","last_became_peered":"2024-03-24T11:39:24.323056+0000","last_unstale":"2024-03-24T11:40:00.374687+0000","last_undegraded":"2024-03-24T11:40:00.374687+0000","last_fullsized":"2024-03-24T11:40:00.374687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:47.808051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376610+0000","last_change":"2024-03-24T11:39:26.340801+0000","last_active":"2024-03-24T11:40:00.376610+0000","last_peered":"2024-03-24T11:40:00.376610+0000","last_clean":"2024-03-24T11:40:00.376610+0000","last_became_active":"2024-03-24T11:39:26.340436+0000","last_became_peered":"2024-03-24T11:39:26.340436+0000","last_unstale":"2024-03-24T11:40:00.376610+0000","last_undegraded":"2024-03-24T11:40:00.376610+0000","last_fullsized":"2024-03-24T11:40:00.376610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:44.979604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482425+0000","last_change":"2024-03-24T11:39:28.356405+0000","last_active":"2024-03-24T11:40:00.482425+0000","last_peered":"2024-03-24T11:40:00.482425+0000","last_clean":"2024-03-24T11:40:00.482425+0000","last_became_active":"2024-03-24T11:39:28.356073+0000","last_became_peered":"2024-03-24T11:39:28.356073+0000","last_unstale":"2024-03-24T11:40:00.482425+0000","last_undegraded":"2024-03-24T11:40:00.482425+0000","last_fullsized":"2024-03-24T11:40:00.482425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:21:52.998338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330527+0000","last_change":"2024-03-24T11:39:26.342944+0000","last_active":"2024-03-24T11:40:42.330527+0000","last_peered":"2024-03-24T11:40:42.330527+0000","last_clean":"2024-03-24T11:40:42.330527+0000","last_became_active":"2024-03-24T11:39:26.340796+0000","last_became_peered":"2024-03-24T11:39:26.340796+0000","last_unstale":"2024-03-24T11:40:42.330527+0000","last_undegraded":"2024-03-24T11:40:42.330527+0000","last_fullsized":"2024-03-24T11:40:42.330527+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:38:35.159323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373582+0000","last_change":"2024-03-24T11:39:22.270253+0000","last_active":"2024-03-24T11:40:00.373582+0000","last_peered":"2024-03-24T11:40:00.373582+0000","last_clean":"2024-03-24T11:40:00.373582+0000","last_became_active":"2024-03-24T11:39:22.269983+0000","last_became_peered":"2024-03-24T11:39:22.269983+0000","last_unstale":"2024-03-24T11:40:00.373582+0000","last_undegraded":"2024-03-24T11:40:00.373582+0000","last_fullsized":"2024-03-24T11:40:00.373582+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:08:57.543296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":29,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374871+0000","last_change":"2024-03-24T11:39:48.140282+0000","last_active":"2024-03-24T11:40:00.374871+0000","last_peered":"2024-03-24T11:40:00.374871+0000","last_clean":"2024-03-24T11:40:00.374871+0000","last_became_active":"2024-03-24T11:39:48.126374+0000","last_became_peered":"2024-03-24T11:39:48.126374+0000","last_unstale":"2024-03-24T11:40:00.374871+0000","last_undegraded":"2024-03-24T11:40:00.374871+0000","last_fullsized":"2024-03-24T11:40:00.374871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:05:42.578844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373834+0000","last_change":"2024-03-24T11:39:29.121297+0000","last_active":"2024-03-24T11:40:00.373834+0000","last_peered":"2024-03-24T11:40:00.373834+0000","last_clean":"2024-03-24T11:40:00.373834+0000","last_became_active":"2024-03-24T11:39:29.120896+0000","last_became_peered":"2024-03-24T11:39:29.120896+0000","last_unstale":"2024-03-24T11:40:00.373834+0000","last_undegraded":"2024-03-24T11:40:00.373834+0000","last_fullsized":"2024-03-24T11:40:00.373834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:54.731060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482776+0000","last_change":"2024-03-24T11:39:26.344752+0000","last_active":"2024-03-24T11:40:00.482776+0000","last_peered":"2024-03-24T11:40:00.482776+0000","last_clean":"2024-03-24T11:40:00.482776+0000","last_became_active":"2024-03-24T11:39:26.344019+0000","last_became_peered":"2024-03-24T11:39:26.344019+0000","last_unstale":"2024-03-24T11:40:00.482776+0000","last_undegraded":"2024-03-24T11:40:00.482776+0000","last_fullsized":"2024-03-24T11:40:00.482776+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:13:36.517702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376696+0000","last_change":"2024-03-24T11:39:22.271959+0000","last_active":"2024-03-24T11:40:00.376696+0000","last_peered":"2024-03-24T11:40:00.376696+0000","last_clean":"2024-03-24T11:40:00.376696+0000","last_became_active":"2024-03-24T11:39:22.271671+0000","last_became_peered":"2024-03-24T11:39:22.271671+0000","last_unstale":"2024-03-24T11:40:00.376696+0000","last_undegraded":"2024-03-24T11:40:00.376696+0000","last_fullsized":"2024-03-24T11:40:00.376696+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:48:00.296407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377792+0000","last_change":"2024-03-24T11:39:24.326366+0000","last_active":"2024-03-24T11:40:00.377792+0000","last_peered":"2024-03-24T11:40:00.377792+0000","last_clean":"2024-03-24T11:40:00.377792+0000","last_became_active":"2024-03-24T11:39:24.325774+0000","last_became_peered":"2024-03-24T11:39:24.325774+0000","last_unstale":"2024-03-24T11:40:00.377792+0000","last_undegraded":"2024-03-24T11:40:00.377792+0000","last_fullsized":"2024-03-24T11:40:00.377792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:48.678848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375019+0000","last_change":"2024-03-24T11:39:28.354215+0000","last_active":"2024-03-24T11:40:00.375019+0000","last_peered":"2024-03-24T11:40:00.375019+0000","last_clean":"2024-03-24T11:40:00.375019+0000","last_became_active":"2024-03-24T11:39:28.353929+0000","last_became_peered":"2024-03-24T11:39:28.353929+0000","last_unstale":"2024-03-24T11:40:00.375019+0000","last_undegraded":"2024-03-24T11:40:00.375019+0000","last_fullsized":"2024-03-24T11:40:00.375019+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:28:01.666879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215005+0000","last_change":"2024-03-24T11:39:26.342730+0000","last_active":"2024-03-24T11:40:01.215005+0000","last_peered":"2024-03-24T11:40:01.215005+0000","last_clean":"2024-03-24T11:40:01.215005+0000","last_became_active":"2024-03-24T11:39:26.341904+0000","last_became_peered":"2024-03-24T11:39:26.341904+0000","last_unstale":"2024-03-24T11:40:01.215005+0000","last_undegraded":"2024-03-24T11:40:01.215005+0000","last_fullsized":"2024-03-24T11:40:01.215005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:34:17.492295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112688+0000","last_change":"2024-03-24T11:39:22.274718+0000","last_active":"2024-03-24T11:40:01.112688+0000","last_peered":"2024-03-24T11:40:01.112688+0000","last_clean":"2024-03-24T11:40:01.112688+0000","last_became_active":"2024-03-24T11:39:22.273663+0000","last_became_peered":"2024-03-24T11:39:22.273663+0000","last_unstale":"2024-03-24T11:40:01.112688+0000","last_undegraded":"2024-03-24T11:40:01.112688+0000","last_fullsized":"2024-03-24T11:40:01.112688+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:37.272429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377372+0000","last_change":"2024-03-24T11:39:24.327870+0000","last_active":"2024-03-24T11:40:00.377372+0000","last_peered":"2024-03-24T11:40:00.377372+0000","last_clean":"2024-03-24T11:40:00.377372+0000","last_became_active":"2024-03-24T11:39:24.327329+0000","last_became_peered":"2024-03-24T11:39:24.327329+0000","last_unstale":"2024-03-24T11:40:00.377372+0000","last_undegraded":"2024-03-24T11:40:00.377372+0000","last_fullsized":"2024-03-24T11:40:00.377372+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:18:24.755060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378136+0000","last_change":"2024-03-24T11:39:28.350475+0000","last_active":"2024-03-24T11:40:00.378136+0000","last_peered":"2024-03-24T11:40:00.378136+0000","last_clean":"2024-03-24T11:40:00.378136+0000","last_became_active":"2024-03-24T11:39:28.350208+0000","last_became_peered":"2024-03-24T11:39:28.350208+0000","last_unstale":"2024-03-24T11:40:00.378136+0000","last_undegraded":"2024-03-24T11:40:00.378136+0000","last_fullsized":"2024-03-24T11:40:00.378136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:24:20.858087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330363+0000","last_change":"2024-03-24T11:39:26.344831+0000","last_active":"2024-03-24T11:40:42.330363+0000","last_peered":"2024-03-24T11:40:42.330363+0000","last_clean":"2024-03-24T11:40:42.330363+0000","last_became_active":"2024-03-24T11:39:26.344445+0000","last_became_peered":"2024-03-24T11:39:26.344445+0000","last_unstale":"2024-03-24T11:40:42.330363+0000","last_undegraded":"2024-03-24T11:40:42.330363+0000","last_fullsized":"2024-03-24T11:40:42.330363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:11:52.503048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373673+0000","last_change":"2024-03-24T11:39:22.271027+0000","last_active":"2024-03-24T11:40:00.373673+0000","last_peered":"2024-03-24T11:40:00.373673+0000","last_clean":"2024-03-24T11:40:00.373673+0000","last_became_active":"2024-03-24T11:39:22.270729+0000","last_became_peered":"2024-03-24T11:39:22.270729+0000","last_unstale":"2024-03-24T11:40:00.373673+0000","last_undegraded":"2024-03-24T11:40:00.373673+0000","last_fullsized":"2024-03-24T11:40:00.373673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:25:54.270435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378060+0000","last_change":"2024-03-24T11:39:24.323492+0000","last_active":"2024-03-24T11:40:00.378060+0000","last_peered":"2024-03-24T11:40:00.378060+0000","last_clean":"2024-03-24T11:40:00.378060+0000","last_became_active":"2024-03-24T11:39:24.323175+0000","last_became_peered":"2024-03-24T11:39:24.323175+0000","last_unstale":"2024-03-24T11:40:00.378060+0000","last_undegraded":"2024-03-24T11:40:00.378060+0000","last_fullsized":"2024-03-24T11:40:00.378060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:31.912447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201302,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1310720,"data_stored":869990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462940,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1286144,"data_stored":868925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":745472,"data_stored":281065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1314816,"data_stored":870117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":716800,"data_stored":280110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":279448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":729088,"data_stored":279573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724577,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":700416,"data_stored":279758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:40:53.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:40:54.165 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:54.566 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:54 smithi114 ceph-mon[30312]: from='client.24640 v1:172.21.15.114:0/1857313308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:40:54.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:54 smithi114 ceph-mon[42466]: from='client.24640 v1:172.21.15.114:0/1857313308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:40:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:54 smithi119 ceph-mon[33714]: from='client.24640 v1:172.21.15.114:0/1857313308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:40:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:40:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:40:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:55.256 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:54 smithi119 bash[66701]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:40:55.257 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 bash[66701]: Writing manifest to image destination 2024-03-24T11:40:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-mon[33714]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:55.554 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 podman[66701]: 2024-03-24 11:40:55.231687517 +0000 UTC m=+2.304060394 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:40:55.554 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 podman[66701]: 2024-03-24 11:40:55.363820695 +0000 UTC m=+2.436193599 container create dc2d3e8ddd15b81d19a81b64ac15ae1f97ed60647b1caf519c505204870625ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:55.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:55 smithi114 ceph-mon[30312]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:55.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:55 smithi114 ceph-mon[42466]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:55.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.864+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/2608952038 >> v1:172.21.15.114:6789/0 conn(0x7ff3d810e4a0 legacy=0x7ff3d8110960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:55.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.865+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/2608952038 shutdown_connections 2024-03-24T11:40:55.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.865+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/2608952038 >> 172.21.15.114:0/2608952038 conn(0x7ff3d80fd510 msgr2=0x7ff3d80ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:55.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.865+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/2608952038 shutdown_connections 2024-03-24T11:40:55.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.865+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/2608952038 wait complete. 2024-03-24T11:40:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.865+0000 7ff3dcc1d700 1 Processor -- start 2024-03-24T11:40:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.865+0000 7ff3dcc1d700 1 -- start start 2024-03-24T11:40:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3dcc1d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff3d81a8da0 con 0x7ff3d810a8d0 2024-03-24T11:40:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3dcc1d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff3d81a9f80 con 0x7ff3d8107a20 2024-03-24T11:40:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3dcc1d700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff3d81ab180 con 0x7ff3d810e4a0 2024-03-24T11:40:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3d6d9d700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff3d810e4a0 0x7ff3d81a74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39386/0 (socket says 172.21.15.114:39386) 2024-03-24T11:40:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3d5d9b700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff3d810a8d0 0x7ff3d81a3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39082/0 (socket says 172.21.15.114:39082) 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3d6d9d700 1 -- 172.21.15.114:0/1997801331 learned_addr learned my addr 172.21.15.114:0/1997801331 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970512158 0 0) 0x7ff3d81a8da0 con 0x7ff3d810a8d0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c00035b0 con 0x7ff3d810a8d0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998765458 0 0) 0x7ff3d81ab180 con 0x7ff3d810e4a0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d81a8da0 con 0x7ff3d810e4a0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825567509 0 0) 0x7ff3d81a9f80 con 0x7ff3d8107a20 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d81ab180 con 0x7ff3d8107a20 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.866+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2191870140 0 0) 0x7ff3d81a8da0 con 0x7ff3d810e4a0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d81a9f80 con 0x7ff3d810e4a0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3059362647 0 0) 0x7ff3c00035b0 con 0x7ff3d810a8d0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d81a8da0 con 0x7ff3d810a8d0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff3d0001c00 con 0x7ff3d810e4a0 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3978772427 0 0) 0x7ff3d81ab180 con 0x7ff3d8107a20 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3c00035b0 con 0x7ff3d8107a20 2024-03-24T11:40:55.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff3cc0017f0 con 0x7ff3d810a8d0 2024-03-24T11:40:55.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff3c8001850 con 0x7ff3d8107a20 2024-03-24T11:40:55.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 200343928 0 0) 0x7ff3d81a9f80 con 0x7ff3d810e4a0 2024-03-24T11:40:55.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 >> v1:172.21.15.119:6789/0 conn(0x7ff3d8107a20 legacy=0x7ff3d80758b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:55.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 >> v1:172.21.15.114:6789/0 conn(0x7ff3d810a8d0 legacy=0x7ff3d81a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d81ac380 con 0x7ff3d810e4a0 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d81a8f90 con 0x7ff3d810e4a0 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d81a9410 con 0x7ff3d810e4a0 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ff3d0004a50 con 0x7ff3d810e4a0 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.867+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ff3d0005290 con 0x7ff3d810e4a0 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.868+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d8103160 con 0x7ff3d810e4a0 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.869+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7ff3d00054d0 con 0x7ff3d810e4a0 2024-03-24T11:40:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.869+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7ff3d00888a0 con 0x7ff3d810e4a0 2024-03-24T11:40:55.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:55.874+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ff3d0055de0 con 0x7ff3d810e4a0 2024-03-24T11:40:56.054 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 podman[66701]: 2024-03-24 11:40:55.63281318 +0000 UTC m=+2.705186051 container init dc2d3e8ddd15b81d19a81b64ac15ae1f97ed60647b1caf519c505204870625ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:56.054 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 podman[66701]: 2024-03-24 11:40:55.638120707 +0000 UTC m=+2.710493568 container start dc2d3e8ddd15b81d19a81b64ac15ae1f97ed60647b1caf519c505204870625ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:56.054 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.641Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.641Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.642Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.642Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.642Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:40:56.055 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:40:56.056 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:40:56.057 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:40:56.058 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:40:56.058 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:40:56.058 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:40:56.059 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:40:56.059 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:40:56.059 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:40:56.059 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:40:56.059 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:40:56.059 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:40:56.059 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:40:56.060 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:40:56.060 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:40:56.060 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:40:56.060 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b[66751]: ts=2024-03-24T11:40:55.643Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:40:56.060 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 bash[66701]: dc2d3e8ddd15b81d19a81b64ac15ae1f97ed60647b1caf519c505204870625ef 2024-03-24T11:40:56.060 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 11:40:55 smithi119 systemd[1]: Started Ceph node-exporter.b for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:40:56.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.096+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 --> v1:172.21.15.114:6800/1253397276 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3d8114830 con 0x7ff3c006e6f0 2024-03-24T11:40:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.105+0000 7ff3c77fe700 1 -- 172.21.15.114:0/1997801331 <== mgr.14589 v1:172.21.15.114:6800/1253397276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346475 (unknown 2965378022 0 4230923120) 0x7ff3c8001c70 con 0x7ff3c006e6f0 2024-03-24T11:40:56.111 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:40:56.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.109+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 >> v1:172.21.15.114:6800/1253397276 conn(0x7ff3c006e6f0 legacy=0x7ff3c0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:56.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.109+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 >> v1:172.21.15.114:6790/0 conn(0x7ff3d810e4a0 legacy=0x7ff3d81a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:56.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.109+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 shutdown_connections 2024-03-24T11:40:56.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.109+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 >> 172.21.15.114:0/1997801331 conn(0x7ff3d80fd510 msgr2=0x7ff3d8101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:56.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.109+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 shutdown_connections 2024-03-24T11:40:56.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:56.109+0000 7ff3dcc1d700 1 -- 172.21.15.114:0/1997801331 wait complete. 2024-03-24T11:40:56.121 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-24T11:40:56.914 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-24T11:40:54.396238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1016,"num_read_kb":862,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235992,"kb_used_data":7356,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725224,"statfs":{"total":767960285184,"available":767718629376,"internally_reserved":0,"allocated":7532544,"data_stored":4008986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002947"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376185+0000","last_change":"2024-03-24T11:39:28.352146+0000","last_active":"2024-03-24T11:40:00.376185+0000","last_peered":"2024-03-24T11:40:00.376185+0000","last_clean":"2024-03-24T11:40:00.376185+0000","last_became_active":"2024-03-24T11:39:28.351793+0000","last_became_peered":"2024-03-24T11:39:28.351793+0000","last_unstale":"2024-03-24T11:40:00.376185+0000","last_undegraded":"2024-03-24T11:40:00.376185+0000","last_fullsized":"2024-03-24T11:40:00.376185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:29.116830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214115+0000","last_change":"2024-03-24T11:39:26.341966+0000","last_active":"2024-03-24T11:40:01.214115+0000","last_peered":"2024-03-24T11:40:01.214115+0000","last_clean":"2024-03-24T11:40:01.214115+0000","last_became_active":"2024-03-24T11:39:26.341063+0000","last_became_peered":"2024-03-24T11:39:26.341063+0000","last_unstale":"2024-03-24T11:40:01.214115+0000","last_undegraded":"2024-03-24T11:40:01.214115+0000","last_fullsized":"2024-03-24T11:40:01.214115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:39.315934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215080+0000","last_change":"2024-03-24T11:39:22.275045+0000","last_active":"2024-03-24T11:40:01.215080+0000","last_peered":"2024-03-24T11:40:01.215080+0000","last_clean":"2024-03-24T11:40:01.215080+0000","last_became_active":"2024-03-24T11:39:22.274707+0000","last_became_peered":"2024-03-24T11:39:22.274707+0000","last_unstale":"2024-03-24T11:40:01.215080+0000","last_undegraded":"2024-03-24T11:40:01.215080+0000","last_fullsized":"2024-03-24T11:40:01.215080+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:33.849147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375798+0000","last_change":"2024-03-24T11:39:48.132081+0000","last_active":"2024-03-24T11:40:00.375798+0000","last_peered":"2024-03-24T11:40:00.375798+0000","last_clean":"2024-03-24T11:40:00.375798+0000","last_became_active":"2024-03-24T11:39:48.126605+0000","last_became_peered":"2024-03-24T11:39:48.126605+0000","last_unstale":"2024-03-24T11:40:00.375798+0000","last_undegraded":"2024-03-24T11:40:00.375798+0000","last_fullsized":"2024-03-24T11:40:00.375798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:57:29.208780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373497+0000","last_change":"2024-03-24T11:39:29.121106+0000","last_active":"2024-03-24T11:40:00.373497+0000","last_peered":"2024-03-24T11:40:00.373497+0000","last_clean":"2024-03-24T11:40:00.373497+0000","last_became_active":"2024-03-24T11:39:29.120714+0000","last_became_peered":"2024-03-24T11:39:29.120714+0000","last_unstale":"2024-03-24T11:40:00.373497+0000","last_undegraded":"2024-03-24T11:40:00.373497+0000","last_fullsized":"2024-03-24T11:40:00.373497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:18.037707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375800+0000","last_change":"2024-03-24T11:39:26.342934+0000","last_active":"2024-03-24T11:40:00.375800+0000","last_peered":"2024-03-24T11:40:00.375800+0000","last_clean":"2024-03-24T11:40:00.375800+0000","last_became_active":"2024-03-24T11:39:26.341799+0000","last_became_peered":"2024-03-24T11:39:26.341799+0000","last_unstale":"2024-03-24T11:40:00.375800+0000","last_undegraded":"2024-03-24T11:40:00.375800+0000","last_fullsized":"2024-03-24T11:40:00.375800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:26:13.575390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374566+0000","last_change":"2024-03-24T11:39:22.269836+0000","last_active":"2024-03-24T11:40:00.374566+0000","last_peered":"2024-03-24T11:40:00.374566+0000","last_clean":"2024-03-24T11:40:00.374566+0000","last_became_active":"2024-03-24T11:39:22.269554+0000","last_became_peered":"2024-03-24T11:39:22.269554+0000","last_unstale":"2024-03-24T11:40:00.374566+0000","last_undegraded":"2024-03-24T11:40:00.374566+0000","last_fullsized":"2024-03-24T11:40:00.374566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:42:50.024642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375767+0000","last_change":"2024-03-24T11:39:24.325168+0000","last_active":"2024-03-24T11:40:00.375767+0000","last_peered":"2024-03-24T11:40:00.375767+0000","last_clean":"2024-03-24T11:40:00.375767+0000","last_became_active":"2024-03-24T11:39:24.323915+0000","last_became_peered":"2024-03-24T11:39:24.323915+0000","last_unstale":"2024-03-24T11:40:00.375767+0000","last_undegraded":"2024-03-24T11:40:00.375767+0000","last_fullsized":"2024-03-24T11:40:00.375767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:33.203268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214252+0000","last_change":"2024-03-24T11:39:28.359406+0000","last_active":"2024-03-24T11:40:01.214252+0000","last_peered":"2024-03-24T11:40:01.214252+0000","last_clean":"2024-03-24T11:40:01.214252+0000","last_became_active":"2024-03-24T11:39:28.359087+0000","last_became_peered":"2024-03-24T11:39:28.359087+0000","last_unstale":"2024-03-24T11:40:01.214252+0000","last_undegraded":"2024-03-24T11:40:01.214252+0000","last_fullsized":"2024-03-24T11:40:01.214252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:22.414364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482266+0000","last_change":"2024-03-24T11:39:26.346456+0000","last_active":"2024-03-24T11:40:00.482266+0000","last_peered":"2024-03-24T11:40:00.482266+0000","last_clean":"2024-03-24T11:40:00.482266+0000","last_became_active":"2024-03-24T11:39:26.346107+0000","last_became_peered":"2024-03-24T11:39:26.346107+0000","last_unstale":"2024-03-24T11:40:00.482266+0000","last_undegraded":"2024-03-24T11:40:00.482266+0000","last_fullsized":"2024-03-24T11:40:00.482266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:44.732570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482428+0000","last_change":"2024-03-24T11:39:22.270151+0000","last_active":"2024-03-24T11:40:00.482428+0000","last_peered":"2024-03-24T11:40:00.482428+0000","last_clean":"2024-03-24T11:40:00.482428+0000","last_became_active":"2024-03-24T11:39:22.269862+0000","last_became_peered":"2024-03-24T11:39:22.269862+0000","last_unstale":"2024-03-24T11:40:00.482428+0000","last_undegraded":"2024-03-24T11:40:00.482428+0000","last_fullsized":"2024-03-24T11:40:00.482428+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:53.761052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376412+0000","last_change":"2024-03-24T11:39:24.323561+0000","last_active":"2024-03-24T11:40:00.376412+0000","last_peered":"2024-03-24T11:40:00.376412+0000","last_clean":"2024-03-24T11:40:00.376412+0000","last_became_active":"2024-03-24T11:39:24.323091+0000","last_became_peered":"2024-03-24T11:39:24.323091+0000","last_unstale":"2024-03-24T11:40:00.376412+0000","last_undegraded":"2024-03-24T11:40:00.376412+0000","last_fullsized":"2024-03-24T11:40:00.376412+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:07.730292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482080+0000","last_change":"2024-03-24T11:39:28.352566+0000","last_active":"2024-03-24T11:40:00.482080+0000","last_peered":"2024-03-24T11:40:00.482080+0000","last_clean":"2024-03-24T11:40:00.482080+0000","last_became_active":"2024-03-24T11:39:28.352263+0000","last_became_peered":"2024-03-24T11:39:28.352263+0000","last_unstale":"2024-03-24T11:40:00.482080+0000","last_undegraded":"2024-03-24T11:40:00.482080+0000","last_fullsized":"2024-03-24T11:40:00.482080+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:01:56.522077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497816+0000","last_change":"2024-03-24T11:39:26.341841+0000","last_active":"2024-03-24T11:40:00.497816+0000","last_peered":"2024-03-24T11:40:00.497816+0000","last_clean":"2024-03-24T11:40:00.497816+0000","last_became_active":"2024-03-24T11:39:26.341511+0000","last_became_peered":"2024-03-24T11:39:26.341511+0000","last_unstale":"2024-03-24T11:40:00.497816+0000","last_undegraded":"2024-03-24T11:40:00.497816+0000","last_fullsized":"2024-03-24T11:40:00.497816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:23.921201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482379+0000","last_change":"2024-03-24T11:39:22.276917+0000","last_active":"2024-03-24T11:40:00.482379+0000","last_peered":"2024-03-24T11:40:00.482379+0000","last_clean":"2024-03-24T11:40:00.482379+0000","last_became_active":"2024-03-24T11:39:22.276624+0000","last_became_peered":"2024-03-24T11:39:22.276624+0000","last_unstale":"2024-03-24T11:40:00.482379+0000","last_undegraded":"2024-03-24T11:40:00.482379+0000","last_fullsized":"2024-03-24T11:40:00.482379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:26:48.692979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376417+0000","last_change":"2024-03-24T11:39:24.327759+0000","last_active":"2024-03-24T11:40:00.376417+0000","last_peered":"2024-03-24T11:40:00.376417+0000","last_clean":"2024-03-24T11:40:00.376417+0000","last_became_active":"2024-03-24T11:39:24.327423+0000","last_became_peered":"2024-03-24T11:39:24.327423+0000","last_unstale":"2024-03-24T11:40:00.376417+0000","last_undegraded":"2024-03-24T11:40:00.376417+0000","last_fullsized":"2024-03-24T11:40:00.376417+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:17.824303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377932+0000","last_change":"2024-03-24T11:39:29.120810+0000","last_active":"2024-03-24T11:40:00.377932+0000","last_peered":"2024-03-24T11:40:00.377932+0000","last_clean":"2024-03-24T11:40:00.377932+0000","last_became_active":"2024-03-24T11:39:29.120510+0000","last_became_peered":"2024-03-24T11:39:29.120510+0000","last_unstale":"2024-03-24T11:40:00.377932+0000","last_undegraded":"2024-03-24T11:40:00.377932+0000","last_fullsized":"2024-03-24T11:40:00.377932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:35.044401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215120+0000","last_change":"2024-03-24T11:39:26.342978+0000","last_active":"2024-03-24T11:40:01.215120+0000","last_peered":"2024-03-24T11:40:01.215120+0000","last_clean":"2024-03-24T11:40:01.215120+0000","last_became_active":"2024-03-24T11:39:26.342201+0000","last_became_peered":"2024-03-24T11:39:26.342201+0000","last_unstale":"2024-03-24T11:40:01.215120+0000","last_undegraded":"2024-03-24T11:40:01.215120+0000","last_fullsized":"2024-03-24T11:40:01.215120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:25.363262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.216083+0000","last_change":"2024-03-24T11:39:22.280482+0000","last_active":"2024-03-24T11:40:01.216083+0000","last_peered":"2024-03-24T11:40:01.216083+0000","last_clean":"2024-03-24T11:40:01.216083+0000","last_became_active":"2024-03-24T11:39:22.279980+0000","last_became_peered":"2024-03-24T11:39:22.279980+0000","last_unstale":"2024-03-24T11:40:01.216083+0000","last_undegraded":"2024-03-24T11:40:01.216083+0000","last_fullsized":"2024-03-24T11:40:01.216083+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:35:49.140830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373560+0000","last_change":"2024-03-24T11:39:24.328144+0000","last_active":"2024-03-24T11:40:00.373560+0000","last_peered":"2024-03-24T11:40:00.373560+0000","last_clean":"2024-03-24T11:40:00.373560+0000","last_became_active":"2024-03-24T11:39:24.327800+0000","last_became_peered":"2024-03-24T11:39:24.327800+0000","last_unstale":"2024-03-24T11:40:00.373560+0000","last_undegraded":"2024-03-24T11:40:00.373560+0000","last_fullsized":"2024-03-24T11:40:00.373560+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:05.892896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376313+0000","last_change":"2024-03-24T11:39:29.123532+0000","last_active":"2024-03-24T11:40:00.376313+0000","last_peered":"2024-03-24T11:40:00.376313+0000","last_clean":"2024-03-24T11:40:00.376313+0000","last_became_active":"2024-03-24T11:39:29.123199+0000","last_became_peered":"2024-03-24T11:39:29.123199+0000","last_unstale":"2024-03-24T11:40:00.376313+0000","last_undegraded":"2024-03-24T11:40:00.376313+0000","last_fullsized":"2024-03-24T11:40:00.376313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:38.696993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377962+0000","last_change":"2024-03-24T11:39:26.336886+0000","last_active":"2024-03-24T11:40:00.377962+0000","last_peered":"2024-03-24T11:40:00.377962+0000","last_clean":"2024-03-24T11:40:00.377962+0000","last_became_active":"2024-03-24T11:39:26.336469+0000","last_became_peered":"2024-03-24T11:39:26.336469+0000","last_unstale":"2024-03-24T11:40:00.377962+0000","last_undegraded":"2024-03-24T11:40:00.377962+0000","last_fullsized":"2024-03-24T11:40:00.377962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:11.100949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376521+0000","last_change":"2024-03-24T11:39:22.274217+0000","last_active":"2024-03-24T11:40:00.376521+0000","last_peered":"2024-03-24T11:40:00.376521+0000","last_clean":"2024-03-24T11:40:00.376521+0000","last_became_active":"2024-03-24T11:39:22.273533+0000","last_became_peered":"2024-03-24T11:39:22.273533+0000","last_unstale":"2024-03-24T11:40:00.376521+0000","last_undegraded":"2024-03-24T11:40:00.376521+0000","last_fullsized":"2024-03-24T11:40:00.376521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:42.837194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373537+0000","last_change":"2024-03-24T11:39:24.333633+0000","last_active":"2024-03-24T11:40:00.373537+0000","last_peered":"2024-03-24T11:40:00.373537+0000","last_clean":"2024-03-24T11:40:00.373537+0000","last_became_active":"2024-03-24T11:39:24.333314+0000","last_became_peered":"2024-03-24T11:39:24.333314+0000","last_unstale":"2024-03-24T11:40:00.373537+0000","last_undegraded":"2024-03-24T11:40:00.373537+0000","last_fullsized":"2024-03-24T11:40:00.373537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:41.046356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497963+0000","last_change":"2024-03-24T11:39:28.358462+0000","last_active":"2024-03-24T11:40:00.497963+0000","last_peered":"2024-03-24T11:40:00.497963+0000","last_clean":"2024-03-24T11:40:00.497963+0000","last_became_active":"2024-03-24T11:39:28.358184+0000","last_became_peered":"2024-03-24T11:39:28.358184+0000","last_unstale":"2024-03-24T11:40:00.497963+0000","last_undegraded":"2024-03-24T11:40:00.497963+0000","last_fullsized":"2024-03-24T11:40:00.497963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:37:05.786244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330555+0000","last_change":"2024-03-24T11:39:26.342708+0000","last_active":"2024-03-24T11:40:42.330555+0000","last_peered":"2024-03-24T11:40:42.330555+0000","last_clean":"2024-03-24T11:40:42.330555+0000","last_became_active":"2024-03-24T11:39:26.342382+0000","last_became_peered":"2024-03-24T11:39:26.342382+0000","last_unstale":"2024-03-24T11:40:42.330555+0000","last_undegraded":"2024-03-24T11:40:42.330555+0000","last_fullsized":"2024-03-24T11:40:42.330555+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:47:07.451790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214211+0000","last_change":"2024-03-24T11:39:22.276063+0000","last_active":"2024-03-24T11:40:01.214211+0000","last_peered":"2024-03-24T11:40:01.214211+0000","last_clean":"2024-03-24T11:40:01.214211+0000","last_became_active":"2024-03-24T11:39:22.275726+0000","last_became_peered":"2024-03-24T11:39:22.275726+0000","last_unstale":"2024-03-24T11:40:01.214211+0000","last_undegraded":"2024-03-24T11:40:01.214211+0000","last_fullsized":"2024-03-24T11:40:01.214211+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:05.420375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215358+0000","last_change":"2024-03-24T11:39:24.327617+0000","last_active":"2024-03-24T11:40:01.215358+0000","last_peered":"2024-03-24T11:40:01.215358+0000","last_clean":"2024-03-24T11:40:01.215358+0000","last_became_active":"2024-03-24T11:39:24.327186+0000","last_became_peered":"2024-03-24T11:39:24.327186+0000","last_unstale":"2024-03-24T11:40:01.215358+0000","last_undegraded":"2024-03-24T11:40:01.215358+0000","last_fullsized":"2024-03-24T11:40:01.215358+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:14.082702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215205+0000","last_change":"2024-03-24T11:39:28.350429+0000","last_active":"2024-03-24T11:40:01.215205+0000","last_peered":"2024-03-24T11:40:01.215205+0000","last_clean":"2024-03-24T11:40:01.215205+0000","last_became_active":"2024-03-24T11:39:28.350043+0000","last_became_peered":"2024-03-24T11:39:28.350043+0000","last_unstale":"2024-03-24T11:40:01.215205+0000","last_undegraded":"2024-03-24T11:40:01.215205+0000","last_fullsized":"2024-03-24T11:40:01.215205+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:26:47.934591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215565+0000","last_change":"2024-03-24T11:39:26.340610+0000","last_active":"2024-03-24T11:40:01.215565+0000","last_peered":"2024-03-24T11:40:01.215565+0000","last_clean":"2024-03-24T11:40:01.215565+0000","last_became_active":"2024-03-24T11:39:26.340262+0000","last_became_peered":"2024-03-24T11:39:26.340262+0000","last_unstale":"2024-03-24T11:40:01.215565+0000","last_undegraded":"2024-03-24T11:40:01.215565+0000","last_fullsized":"2024-03-24T11:40:01.215565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:02:04.790525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373359+0000","last_change":"2024-03-24T11:39:22.276032+0000","last_active":"2024-03-24T11:40:00.373359+0000","last_peered":"2024-03-24T11:40:00.373359+0000","last_clean":"2024-03-24T11:40:00.373359+0000","last_became_active":"2024-03-24T11:39:22.275629+0000","last_became_peered":"2024-03-24T11:39:22.275629+0000","last_unstale":"2024-03-24T11:40:00.373359+0000","last_undegraded":"2024-03-24T11:40:00.373359+0000","last_fullsized":"2024-03-24T11:40:00.373359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:10.405957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112961+0000","last_change":"2024-03-24T11:39:24.325862+0000","last_active":"2024-03-24T11:40:01.112961+0000","last_peered":"2024-03-24T11:40:01.112961+0000","last_clean":"2024-03-24T11:40:01.112961+0000","last_became_active":"2024-03-24T11:39:24.325536+0000","last_became_peered":"2024-03-24T11:39:24.325536+0000","last_unstale":"2024-03-24T11:40:01.112961+0000","last_undegraded":"2024-03-24T11:40:01.112961+0000","last_fullsized":"2024-03-24T11:40:01.112961+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:29.444437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497787+0000","last_change":"2024-03-24T11:39:28.355666+0000","last_active":"2024-03-24T11:40:00.497787+0000","last_peered":"2024-03-24T11:40:00.497787+0000","last_clean":"2024-03-24T11:40:00.497787+0000","last_became_active":"2024-03-24T11:39:28.355337+0000","last_became_peered":"2024-03-24T11:39:28.355337+0000","last_unstale":"2024-03-24T11:40:00.497787+0000","last_undegraded":"2024-03-24T11:40:00.497787+0000","last_fullsized":"2024-03-24T11:40:00.497787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:00.651233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112336+0000","last_change":"2024-03-24T11:39:26.344818+0000","last_active":"2024-03-24T11:40:01.112336+0000","last_peered":"2024-03-24T11:40:01.112336+0000","last_clean":"2024-03-24T11:40:01.112336+0000","last_became_active":"2024-03-24T11:39:26.344336+0000","last_became_peered":"2024-03-24T11:39:26.344336+0000","last_unstale":"2024-03-24T11:40:01.112336+0000","last_undegraded":"2024-03-24T11:40:01.112336+0000","last_fullsized":"2024-03-24T11:40:01.112336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:38:15.244685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215224+0000","last_change":"2024-03-24T11:39:22.279784+0000","last_active":"2024-03-24T11:40:01.215224+0000","last_peered":"2024-03-24T11:40:01.215224+0000","last_clean":"2024-03-24T11:40:01.215224+0000","last_became_active":"2024-03-24T11:39:22.279312+0000","last_became_peered":"2024-03-24T11:39:22.279312+0000","last_unstale":"2024-03-24T11:40:01.215224+0000","last_undegraded":"2024-03-24T11:40:01.215224+0000","last_fullsized":"2024-03-24T11:40:01.215224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:33:35.962134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373460+0000","last_change":"2024-03-24T11:39:24.331404+0000","last_active":"2024-03-24T11:40:00.373460+0000","last_peered":"2024-03-24T11:40:00.373460+0000","last_clean":"2024-03-24T11:40:00.373460+0000","last_became_active":"2024-03-24T11:39:24.330164+0000","last_became_peered":"2024-03-24T11:39:24.330164+0000","last_unstale":"2024-03-24T11:40:00.373460+0000","last_undegraded":"2024-03-24T11:40:00.373460+0000","last_fullsized":"2024-03-24T11:40:00.373460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:19:18.995381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373306+0000","last_change":"2024-03-24T11:39:29.121239+0000","last_active":"2024-03-24T11:40:00.373306+0000","last_peered":"2024-03-24T11:40:00.373306+0000","last_clean":"2024-03-24T11:40:00.373306+0000","last_became_active":"2024-03-24T11:39:29.120752+0000","last_became_peered":"2024-03-24T11:39:29.120752+0000","last_unstale":"2024-03-24T11:40:00.373306+0000","last_undegraded":"2024-03-24T11:40:00.373306+0000","last_fullsized":"2024-03-24T11:40:00.373306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:15:11.649187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498035+0000","last_change":"2024-03-24T11:39:26.347323+0000","last_active":"2024-03-24T11:40:00.498035+0000","last_peered":"2024-03-24T11:40:00.498035+0000","last_clean":"2024-03-24T11:40:00.498035+0000","last_became_active":"2024-03-24T11:39:26.347002+0000","last_became_peered":"2024-03-24T11:39:26.347002+0000","last_unstale":"2024-03-24T11:40:00.498035+0000","last_undegraded":"2024-03-24T11:40:00.498035+0000","last_fullsized":"2024-03-24T11:40:00.498035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:55.314835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482561+0000","last_change":"2024-03-24T11:39:22.276924+0000","last_active":"2024-03-24T11:40:00.482561+0000","last_peered":"2024-03-24T11:40:00.482561+0000","last_clean":"2024-03-24T11:40:00.482561+0000","last_became_active":"2024-03-24T11:39:22.276608+0000","last_became_peered":"2024-03-24T11:39:22.276608+0000","last_unstale":"2024-03-24T11:40:00.482561+0000","last_undegraded":"2024-03-24T11:40:00.482561+0000","last_fullsized":"2024-03-24T11:40:00.482561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:30.540917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375637+0000","last_change":"2024-03-24T11:39:24.327208+0000","last_active":"2024-03-24T11:40:00.375637+0000","last_peered":"2024-03-24T11:40:00.375637+0000","last_clean":"2024-03-24T11:40:00.375637+0000","last_became_active":"2024-03-24T11:39:24.326219+0000","last_became_peered":"2024-03-24T11:39:24.326219+0000","last_unstale":"2024-03-24T11:40:00.375637+0000","last_undegraded":"2024-03-24T11:40:00.375637+0000","last_fullsized":"2024-03-24T11:40:00.375637+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:08.619718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214004+0000","last_change":"2024-03-24T11:39:28.359665+0000","last_active":"2024-03-24T11:40:01.214004+0000","last_peered":"2024-03-24T11:40:01.214004+0000","last_clean":"2024-03-24T11:40:01.214004+0000","last_became_active":"2024-03-24T11:39:28.359279+0000","last_became_peered":"2024-03-24T11:39:28.359279+0000","last_unstale":"2024-03-24T11:40:01.214004+0000","last_undegraded":"2024-03-24T11:40:01.214004+0000","last_fullsized":"2024-03-24T11:40:01.214004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:04:25.045915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373588+0000","last_change":"2024-03-24T11:39:26.338454+0000","last_active":"2024-03-24T11:40:00.373588+0000","last_peered":"2024-03-24T11:40:00.373588+0000","last_clean":"2024-03-24T11:40:00.373588+0000","last_became_active":"2024-03-24T11:39:26.338125+0000","last_became_peered":"2024-03-24T11:39:26.338125+0000","last_unstale":"2024-03-24T11:40:00.373588+0000","last_undegraded":"2024-03-24T11:40:00.373588+0000","last_fullsized":"2024-03-24T11:40:00.373588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:47:18.844947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498200+0000","last_change":"2024-03-24T11:39:22.275813+0000","last_active":"2024-03-24T11:40:00.498200+0000","last_peered":"2024-03-24T11:40:00.498200+0000","last_clean":"2024-03-24T11:40:00.498200+0000","last_became_active":"2024-03-24T11:39:22.274712+0000","last_became_peered":"2024-03-24T11:39:22.274712+0000","last_unstale":"2024-03-24T11:40:00.498200+0000","last_undegraded":"2024-03-24T11:40:00.498200+0000","last_fullsized":"2024-03-24T11:40:00.498200+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:20:48.875472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215281+0000","last_change":"2024-03-24T11:39:24.328109+0000","last_active":"2024-03-24T11:40:01.215281+0000","last_peered":"2024-03-24T11:40:01.215281+0000","last_clean":"2024-03-24T11:40:01.215281+0000","last_became_active":"2024-03-24T11:39:24.327638+0000","last_became_peered":"2024-03-24T11:39:24.327638+0000","last_unstale":"2024-03-24T11:40:01.215281+0000","last_undegraded":"2024-03-24T11:40:01.215281+0000","last_fullsized":"2024-03-24T11:40:01.215281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:25:06.857407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375317+0000","last_change":"2024-03-24T11:39:28.361076+0000","last_active":"2024-03-24T11:40:00.375317+0000","last_peered":"2024-03-24T11:40:00.375317+0000","last_clean":"2024-03-24T11:40:00.375317+0000","last_became_active":"2024-03-24T11:39:28.360310+0000","last_became_peered":"2024-03-24T11:39:28.360310+0000","last_unstale":"2024-03-24T11:40:00.375317+0000","last_undegraded":"2024-03-24T11:40:00.375317+0000","last_fullsized":"2024-03-24T11:40:00.375317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:16:05.714030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214732+0000","last_change":"2024-03-24T11:39:26.341302+0000","last_active":"2024-03-24T11:40:01.214732+0000","last_peered":"2024-03-24T11:40:01.214732+0000","last_clean":"2024-03-24T11:40:01.214732+0000","last_became_active":"2024-03-24T11:39:26.340751+0000","last_became_peered":"2024-03-24T11:39:26.340751+0000","last_unstale":"2024-03-24T11:40:01.214732+0000","last_undegraded":"2024-03-24T11:40:01.214732+0000","last_fullsized":"2024-03-24T11:40:01.214732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:29.203219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378229+0000","last_change":"2024-03-24T11:39:22.269859+0000","last_active":"2024-03-24T11:40:00.378229+0000","last_peered":"2024-03-24T11:40:00.378229+0000","last_clean":"2024-03-24T11:40:00.378229+0000","last_became_active":"2024-03-24T11:39:22.269574+0000","last_became_peered":"2024-03-24T11:39:22.269574+0000","last_unstale":"2024-03-24T11:40:00.378229+0000","last_undegraded":"2024-03-24T11:40:00.378229+0000","last_fullsized":"2024-03-24T11:40:00.378229+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:37:03.492581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376234+0000","last_change":"2024-03-24T11:39:24.330079+0000","last_active":"2024-03-24T11:40:00.376234+0000","last_peered":"2024-03-24T11:40:00.376234+0000","last_clean":"2024-03-24T11:40:00.376234+0000","last_became_active":"2024-03-24T11:39:24.329703+0000","last_became_peered":"2024-03-24T11:39:24.329703+0000","last_unstale":"2024-03-24T11:40:00.376234+0000","last_undegraded":"2024-03-24T11:40:00.376234+0000","last_fullsized":"2024-03-24T11:40:00.376234+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:02.144542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215145+0000","last_change":"2024-03-24T11:39:28.361156+0000","last_active":"2024-03-24T11:40:01.215145+0000","last_peered":"2024-03-24T11:40:01.215145+0000","last_clean":"2024-03-24T11:40:01.215145+0000","last_became_active":"2024-03-24T11:39:28.360836+0000","last_became_peered":"2024-03-24T11:39:28.360836+0000","last_unstale":"2024-03-24T11:40:01.215145+0000","last_undegraded":"2024-03-24T11:40:01.215145+0000","last_fullsized":"2024-03-24T11:40:01.215145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:12.265272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330529+0000","last_change":"2024-03-24T11:39:26.341714+0000","last_active":"2024-03-24T11:40:42.330529+0000","last_peered":"2024-03-24T11:40:42.330529+0000","last_clean":"2024-03-24T11:40:42.330529+0000","last_became_active":"2024-03-24T11:39:26.341233+0000","last_became_peered":"2024-03-24T11:39:26.341233+0000","last_unstale":"2024-03-24T11:40:42.330529+0000","last_undegraded":"2024-03-24T11:40:42.330529+0000","last_fullsized":"2024-03-24T11:40:42.330529+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:52.851836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497855+0000","last_change":"2024-03-24T11:39:22.274274+0000","last_active":"2024-03-24T11:40:00.497855+0000","last_peered":"2024-03-24T11:40:00.497855+0000","last_clean":"2024-03-24T11:40:00.497855+0000","last_became_active":"2024-03-24T11:39:22.273733+0000","last_became_peered":"2024-03-24T11:39:22.273733+0000","last_unstale":"2024-03-24T11:40:00.497855+0000","last_undegraded":"2024-03-24T11:40:00.497855+0000","last_fullsized":"2024-03-24T11:40:00.497855+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:34:38.534882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373925+0000","last_change":"2024-03-24T11:39:24.331998+0000","last_active":"2024-03-24T11:40:00.373925+0000","last_peered":"2024-03-24T11:40:00.373925+0000","last_clean":"2024-03-24T11:40:00.373925+0000","last_became_active":"2024-03-24T11:39:24.331694+0000","last_became_peered":"2024-03-24T11:39:24.331694+0000","last_unstale":"2024-03-24T11:40:00.373925+0000","last_undegraded":"2024-03-24T11:40:00.373925+0000","last_fullsized":"2024-03-24T11:40:00.373925+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:24:52.556163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378470+0000","last_change":"2024-03-24T11:39:28.348376+0000","last_active":"2024-03-24T11:40:00.378470+0000","last_peered":"2024-03-24T11:40:00.378470+0000","last_clean":"2024-03-24T11:40:00.378470+0000","last_became_active":"2024-03-24T11:39:28.347911+0000","last_became_peered":"2024-03-24T11:39:28.347911+0000","last_unstale":"2024-03-24T11:40:00.378470+0000","last_undegraded":"2024-03-24T11:40:00.378470+0000","last_fullsized":"2024-03-24T11:40:00.378470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:59:46.740938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330697+0000","last_change":"2024-03-24T11:39:26.340225+0000","last_active":"2024-03-24T11:40:42.330697+0000","last_peered":"2024-03-24T11:40:42.330697+0000","last_clean":"2024-03-24T11:40:42.330697+0000","last_became_active":"2024-03-24T11:39:26.339897+0000","last_became_peered":"2024-03-24T11:39:26.339897+0000","last_unstale":"2024-03-24T11:40:42.330697+0000","last_undegraded":"2024-03-24T11:40:42.330697+0000","last_fullsized":"2024-03-24T11:40:42.330697+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:44:47.566066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214086+0000","last_change":"2024-03-24T11:39:22.278002+0000","last_active":"2024-03-24T11:40:01.214086+0000","last_peered":"2024-03-24T11:40:01.214086+0000","last_clean":"2024-03-24T11:40:01.214086+0000","last_became_active":"2024-03-24T11:39:22.277702+0000","last_became_peered":"2024-03-24T11:39:22.277702+0000","last_unstale":"2024-03-24T11:40:01.214086+0000","last_undegraded":"2024-03-24T11:40:01.214086+0000","last_fullsized":"2024-03-24T11:40:01.214086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:35:18.769871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482769+0000","last_change":"2024-03-24T11:39:24.324557+0000","last_active":"2024-03-24T11:40:00.482769+0000","last_peered":"2024-03-24T11:40:00.482769+0000","last_clean":"2024-03-24T11:40:00.482769+0000","last_became_active":"2024-03-24T11:39:24.324203+0000","last_became_peered":"2024-03-24T11:39:24.324203+0000","last_unstale":"2024-03-24T11:40:00.482769+0000","last_undegraded":"2024-03-24T11:40:00.482769+0000","last_fullsized":"2024-03-24T11:40:00.482769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:04:12.430126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373999+0000","last_change":"2024-03-24T11:39:28.357168+0000","last_active":"2024-03-24T11:40:00.373999+0000","last_peered":"2024-03-24T11:40:00.373999+0000","last_clean":"2024-03-24T11:40:00.373999+0000","last_became_active":"2024-03-24T11:39:28.356800+0000","last_became_peered":"2024-03-24T11:39:28.356800+0000","last_unstale":"2024-03-24T11:40:00.373999+0000","last_undegraded":"2024-03-24T11:40:00.373999+0000","last_fullsized":"2024-03-24T11:40:00.373999+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:43:55.200971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376231+0000","last_change":"2024-03-24T11:39:26.339993+0000","last_active":"2024-03-24T11:40:00.376231+0000","last_peered":"2024-03-24T11:40:00.376231+0000","last_clean":"2024-03-24T11:40:00.376231+0000","last_became_active":"2024-03-24T11:39:26.339644+0000","last_became_peered":"2024-03-24T11:39:26.339644+0000","last_unstale":"2024-03-24T11:40:00.376231+0000","last_undegraded":"2024-03-24T11:40:00.376231+0000","last_fullsized":"2024-03-24T11:40:00.376231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:00:14.664377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497813+0000","last_change":"2024-03-24T11:39:22.274140+0000","last_active":"2024-03-24T11:40:00.497813+0000","last_peered":"2024-03-24T11:40:00.497813+0000","last_clean":"2024-03-24T11:40:00.497813+0000","last_became_active":"2024-03-24T11:39:22.273798+0000","last_became_peered":"2024-03-24T11:39:22.273798+0000","last_unstale":"2024-03-24T11:40:00.497813+0000","last_undegraded":"2024-03-24T11:40:00.497813+0000","last_fullsized":"2024-03-24T11:40:00.497813+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:14:22.212219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215442+0000","last_change":"2024-03-24T11:39:24.324454+0000","last_active":"2024-03-24T11:40:01.215442+0000","last_peered":"2024-03-24T11:40:01.215442+0000","last_clean":"2024-03-24T11:40:01.215442+0000","last_became_active":"2024-03-24T11:39:24.324115+0000","last_became_peered":"2024-03-24T11:39:24.324115+0000","last_unstale":"2024-03-24T11:40:01.215442+0000","last_undegraded":"2024-03-24T11:40:01.215442+0000","last_fullsized":"2024-03-24T11:40:01.215442+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:37:32.036427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498166+0000","last_change":"2024-03-24T11:39:29.121414+0000","last_active":"2024-03-24T11:40:00.498166+0000","last_peered":"2024-03-24T11:40:00.498166+0000","last_clean":"2024-03-24T11:40:00.498166+0000","last_became_active":"2024-03-24T11:39:29.121062+0000","last_became_peered":"2024-03-24T11:39:29.121062+0000","last_unstale":"2024-03-24T11:40:00.498166+0000","last_undegraded":"2024-03-24T11:40:00.498166+0000","last_fullsized":"2024-03-24T11:40:00.498166+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:13.679597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482607+0000","last_change":"2024-03-24T11:39:26.339373+0000","last_active":"2024-03-24T11:40:00.482607+0000","last_peered":"2024-03-24T11:40:00.482607+0000","last_clean":"2024-03-24T11:40:00.482607+0000","last_became_active":"2024-03-24T11:39:26.338904+0000","last_became_peered":"2024-03-24T11:39:26.338904+0000","last_unstale":"2024-03-24T11:40:00.482607+0000","last_undegraded":"2024-03-24T11:40:00.482607+0000","last_fullsized":"2024-03-24T11:40:00.482607+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:46:47.552299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112318+0000","last_change":"2024-03-24T11:39:22.274494+0000","last_active":"2024-03-24T11:40:01.112318+0000","last_peered":"2024-03-24T11:40:01.112318+0000","last_clean":"2024-03-24T11:40:01.112318+0000","last_became_active":"2024-03-24T11:39:22.274217+0000","last_became_peered":"2024-03-24T11:39:22.274217+0000","last_unstale":"2024-03-24T11:40:01.112318+0000","last_undegraded":"2024-03-24T11:40:01.112318+0000","last_fullsized":"2024-03-24T11:40:01.112318+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:33:12.203983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376399+0000","last_change":"2024-03-24T11:39:24.333126+0000","last_active":"2024-03-24T11:40:00.376399+0000","last_peered":"2024-03-24T11:40:00.376399+0000","last_clean":"2024-03-24T11:40:00.376399+0000","last_became_active":"2024-03-24T11:39:24.332828+0000","last_became_peered":"2024-03-24T11:39:24.332828+0000","last_unstale":"2024-03-24T11:40:00.376399+0000","last_undegraded":"2024-03-24T11:40:00.376399+0000","last_fullsized":"2024-03-24T11:40:00.376399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:30:49.938650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482323+0000","last_change":"2024-03-24T11:39:29.121002+0000","last_active":"2024-03-24T11:40:00.482323+0000","last_peered":"2024-03-24T11:40:00.482323+0000","last_clean":"2024-03-24T11:40:00.482323+0000","last_became_active":"2024-03-24T11:39:29.120720+0000","last_became_peered":"2024-03-24T11:39:29.120720+0000","last_unstale":"2024-03-24T11:40:00.482323+0000","last_undegraded":"2024-03-24T11:40:00.482323+0000","last_fullsized":"2024-03-24T11:40:00.482323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:28:20.422181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498419+0000","last_change":"2024-03-24T11:39:22.279264+0000","last_active":"2024-03-24T11:40:00.498419+0000","last_peered":"2024-03-24T11:40:00.498419+0000","last_clean":"2024-03-24T11:40:00.498419+0000","last_became_active":"2024-03-24T11:39:22.278908+0000","last_became_peered":"2024-03-24T11:39:22.278908+0000","last_unstale":"2024-03-24T11:40:00.498419+0000","last_undegraded":"2024-03-24T11:40:00.498419+0000","last_fullsized":"2024-03-24T11:40:00.498419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:52:18.918085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482092+0000","last_change":"2024-03-24T11:39:24.318820+0000","last_active":"2024-03-24T11:40:00.482092+0000","last_peered":"2024-03-24T11:40:00.482092+0000","last_clean":"2024-03-24T11:40:00.482092+0000","last_became_active":"2024-03-24T11:39:24.318488+0000","last_became_peered":"2024-03-24T11:39:24.318488+0000","last_unstale":"2024-03-24T11:40:00.482092+0000","last_undegraded":"2024-03-24T11:40:00.482092+0000","last_fullsized":"2024-03-24T11:40:00.482092+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:50:11.994317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330790+0000","last_change":"2024-03-24T11:39:26.342727+0000","last_active":"2024-03-24T11:40:42.330790+0000","last_peered":"2024-03-24T11:40:42.330790+0000","last_clean":"2024-03-24T11:40:42.330790+0000","last_became_active":"2024-03-24T11:39:26.342201+0000","last_became_peered":"2024-03-24T11:39:26.342201+0000","last_unstale":"2024-03-24T11:40:42.330790+0000","last_undegraded":"2024-03-24T11:40:42.330790+0000","last_fullsized":"2024-03-24T11:40:42.330790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:38.176799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373407+0000","last_change":"2024-03-24T11:39:28.357336+0000","last_active":"2024-03-24T11:40:00.373407+0000","last_peered":"2024-03-24T11:40:00.373407+0000","last_clean":"2024-03-24T11:40:00.373407+0000","last_became_active":"2024-03-24T11:39:28.357024+0000","last_became_peered":"2024-03-24T11:39:28.357024+0000","last_unstale":"2024-03-24T11:40:00.373407+0000","last_undegraded":"2024-03-24T11:40:00.373407+0000","last_fullsized":"2024-03-24T11:40:00.373407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:20.869919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498352+0000","last_change":"2024-03-24T11:39:22.279163+0000","last_active":"2024-03-24T11:40:00.498352+0000","last_peered":"2024-03-24T11:40:00.498352+0000","last_clean":"2024-03-24T11:40:00.498352+0000","last_became_active":"2024-03-24T11:39:22.278855+0000","last_became_peered":"2024-03-24T11:39:22.278855+0000","last_unstale":"2024-03-24T11:40:00.498352+0000","last_undegraded":"2024-03-24T11:40:00.498352+0000","last_fullsized":"2024-03-24T11:40:00.498352+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:09.826860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377031+0000","last_change":"2024-03-24T11:39:24.329936+0000","last_active":"2024-03-24T11:40:00.377031+0000","last_peered":"2024-03-24T11:40:00.377031+0000","last_clean":"2024-03-24T11:40:00.377031+0000","last_became_active":"2024-03-24T11:39:24.329578+0000","last_became_peered":"2024-03-24T11:39:24.329578+0000","last_unstale":"2024-03-24T11:40:00.377031+0000","last_undegraded":"2024-03-24T11:40:00.377031+0000","last_fullsized":"2024-03-24T11:40:00.377031+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:17:51.128016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373035+0000","last_change":"2024-03-24T11:39:26.338031+0000","last_active":"2024-03-24T11:40:00.373035+0000","last_peered":"2024-03-24T11:40:00.373035+0000","last_clean":"2024-03-24T11:40:00.373035+0000","last_became_active":"2024-03-24T11:39:26.337661+0000","last_became_peered":"2024-03-24T11:39:26.337661+0000","last_unstale":"2024-03-24T11:40:00.373035+0000","last_undegraded":"2024-03-24T11:40:00.373035+0000","last_fullsized":"2024-03-24T11:40:00.373035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:03.905497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.497650+0000","last_change":"2024-03-24T11:39:28.355496+0000","last_active":"2024-03-24T11:40:00.497650+0000","last_peered":"2024-03-24T11:40:00.497650+0000","last_clean":"2024-03-24T11:40:00.497650+0000","last_became_active":"2024-03-24T11:39:28.355197+0000","last_became_peered":"2024-03-24T11:39:28.355197+0000","last_unstale":"2024-03-24T11:40:00.497650+0000","last_undegraded":"2024-03-24T11:40:00.497650+0000","last_fullsized":"2024-03-24T11:40:00.497650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:42.142473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373386+0000","last_change":"2024-03-24T11:39:26.339649+0000","last_active":"2024-03-24T11:40:00.373386+0000","last_peered":"2024-03-24T11:40:00.373386+0000","last_clean":"2024-03-24T11:40:00.373386+0000","last_became_active":"2024-03-24T11:39:26.339188+0000","last_became_peered":"2024-03-24T11:39:26.339188+0000","last_unstale":"2024-03-24T11:40:00.373386+0000","last_undegraded":"2024-03-24T11:40:00.373386+0000","last_fullsized":"2024-03-24T11:40:00.373386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:50:18.916718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498307+0000","last_change":"2024-03-24T11:39:22.272264+0000","last_active":"2024-03-24T11:40:00.498307+0000","last_peered":"2024-03-24T11:40:00.498307+0000","last_clean":"2024-03-24T11:40:00.498307+0000","last_became_active":"2024-03-24T11:39:22.271945+0000","last_became_peered":"2024-03-24T11:39:22.271945+0000","last_unstale":"2024-03-24T11:40:00.498307+0000","last_undegraded":"2024-03-24T11:40:00.498307+0000","last_fullsized":"2024-03-24T11:40:00.498307+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:50.400313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112758+0000","last_change":"2024-03-24T11:39:24.326143+0000","last_active":"2024-03-24T11:40:01.112758+0000","last_peered":"2024-03-24T11:40:01.112758+0000","last_clean":"2024-03-24T11:40:01.112758+0000","last_became_active":"2024-03-24T11:39:24.325479+0000","last_became_peered":"2024-03-24T11:39:24.325479+0000","last_unstale":"2024-03-24T11:40:01.112758+0000","last_undegraded":"2024-03-24T11:40:01.112758+0000","last_fullsized":"2024-03-24T11:40:01.112758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:02:58.851736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214026+0000","last_change":"2024-03-24T11:39:28.357405+0000","last_active":"2024-03-24T11:40:01.214026+0000","last_peered":"2024-03-24T11:40:01.214026+0000","last_clean":"2024-03-24T11:40:01.214026+0000","last_became_active":"2024-03-24T11:39:28.357093+0000","last_became_peered":"2024-03-24T11:39:28.357093+0000","last_unstale":"2024-03-24T11:40:01.214026+0000","last_undegraded":"2024-03-24T11:40:01.214026+0000","last_fullsized":"2024-03-24T11:40:01.214026+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:35:43.876501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373177+0000","last_change":"2024-03-24T11:39:26.333304+0000","last_active":"2024-03-24T11:40:00.373177+0000","last_peered":"2024-03-24T11:40:00.373177+0000","last_clean":"2024-03-24T11:40:00.373177+0000","last_became_active":"2024-03-24T11:39:26.332971+0000","last_became_peered":"2024-03-24T11:39:26.332971+0000","last_unstale":"2024-03-24T11:40:00.373177+0000","last_undegraded":"2024-03-24T11:40:00.373177+0000","last_fullsized":"2024-03-24T11:40:00.373177+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:39:11.165619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482417+0000","last_change":"2024-03-24T11:39:22.270901+0000","last_active":"2024-03-24T11:40:00.482417+0000","last_peered":"2024-03-24T11:40:00.482417+0000","last_clean":"2024-03-24T11:40:00.482417+0000","last_became_active":"2024-03-24T11:39:22.270402+0000","last_became_peered":"2024-03-24T11:39:22.270402+0000","last_unstale":"2024-03-24T11:40:00.482417+0000","last_undegraded":"2024-03-24T11:40:00.482417+0000","last_fullsized":"2024-03-24T11:40:00.482417+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:59:17.729541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215800+0000","last_change":"2024-03-24T11:39:24.329920+0000","last_active":"2024-03-24T11:40:01.215800+0000","last_peered":"2024-03-24T11:40:01.215800+0000","last_clean":"2024-03-24T11:40:01.215800+0000","last_became_active":"2024-03-24T11:39:24.329567+0000","last_became_peered":"2024-03-24T11:39:24.329567+0000","last_unstale":"2024-03-24T11:40:01.215800+0000","last_undegraded":"2024-03-24T11:40:01.215800+0000","last_fullsized":"2024-03-24T11:40:01.215800+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:11:52.558860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374284+0000","last_change":"2024-03-24T11:39:28.358245+0000","last_active":"2024-03-24T11:40:00.374284+0000","last_peered":"2024-03-24T11:40:00.374284+0000","last_clean":"2024-03-24T11:40:00.374284+0000","last_became_active":"2024-03-24T11:39:28.357926+0000","last_became_peered":"2024-03-24T11:39:28.357926+0000","last_unstale":"2024-03-24T11:40:00.374284+0000","last_undegraded":"2024-03-24T11:40:00.374284+0000","last_fullsized":"2024-03-24T11:40:00.374284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:52:35.025907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482136+0000","last_change":"2024-03-24T11:39:26.338093+0000","last_active":"2024-03-24T11:40:00.482136+0000","last_peered":"2024-03-24T11:40:00.482136+0000","last_clean":"2024-03-24T11:40:00.482136+0000","last_became_active":"2024-03-24T11:39:22.276126+0000","last_became_peered":"2024-03-24T11:39:22.276126+0000","last_unstale":"2024-03-24T11:40:00.482136+0000","last_undegraded":"2024-03-24T11:40:00.482136+0000","last_fullsized":"2024-03-24T11:40:00.482136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:08.764768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050133859999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376017+0000","last_change":"2024-03-24T11:39:22.278162+0000","last_active":"2024-03-24T11:40:00.376017+0000","last_peered":"2024-03-24T11:40:00.376017+0000","last_clean":"2024-03-24T11:40:00.376017+0000","last_became_active":"2024-03-24T11:39:22.274727+0000","last_became_peered":"2024-03-24T11:39:22.274727+0000","last_unstale":"2024-03-24T11:40:00.376017+0000","last_undegraded":"2024-03-24T11:40:00.376017+0000","last_fullsized":"2024-03-24T11:40:00.376017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:15:18.230675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":95,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330366+0000","last_change":"2024-03-24T11:39:24.334604+0000","last_active":"2024-03-24T11:40:42.330366+0000","last_peered":"2024-03-24T11:40:42.330366+0000","last_clean":"2024-03-24T11:40:42.330366+0000","last_became_active":"2024-03-24T11:39:24.334273+0000","last_became_peered":"2024-03-24T11:39:24.334273+0000","last_unstale":"2024-03-24T11:40:42.330366+0000","last_undegraded":"2024-03-24T11:40:42.330366+0000","last_fullsized":"2024-03-24T11:40:42.330366+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:27.778605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214808+0000","last_change":"2024-03-24T11:39:26.343791+0000","last_active":"2024-03-24T11:40:01.214808+0000","last_peered":"2024-03-24T11:40:01.214808+0000","last_clean":"2024-03-24T11:40:01.214808+0000","last_became_active":"2024-03-24T11:39:26.342830+0000","last_became_peered":"2024-03-24T11:39:26.342830+0000","last_unstale":"2024-03-24T11:40:01.214808+0000","last_undegraded":"2024-03-24T11:40:01.214808+0000","last_fullsized":"2024-03-24T11:40:01.214808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:55.962463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498399+0000","last_change":"2024-03-24T11:39:29.121058+0000","last_active":"2024-03-24T11:40:00.498399+0000","last_peered":"2024-03-24T11:40:00.498399+0000","last_clean":"2024-03-24T11:40:00.498399+0000","last_became_active":"2024-03-24T11:39:29.120626+0000","last_became_peered":"2024-03-24T11:39:29.120626+0000","last_unstale":"2024-03-24T11:40:00.498399+0000","last_undegraded":"2024-03-24T11:40:00.498399+0000","last_fullsized":"2024-03-24T11:40:00.498399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:52:46.205011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214627+0000","last_change":"2024-03-24T11:39:22.271768+0000","last_active":"2024-03-24T11:40:01.214627+0000","last_peered":"2024-03-24T11:40:01.214627+0000","last_clean":"2024-03-24T11:40:01.214627+0000","last_became_active":"2024-03-24T11:39:22.271342+0000","last_became_peered":"2024-03-24T11:39:22.271342+0000","last_unstale":"2024-03-24T11:40:01.214627+0000","last_undegraded":"2024-03-24T11:40:01.214627+0000","last_fullsized":"2024-03-24T11:40:01.214627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:01:03.354295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373079+0000","last_change":"2024-03-24T11:39:26.331241+0000","last_active":"2024-03-24T11:40:00.373079+0000","last_peered":"2024-03-24T11:40:00.373079+0000","last_clean":"2024-03-24T11:40:00.373079+0000","last_became_active":"2024-03-24T11:39:22.270767+0000","last_became_peered":"2024-03-24T11:39:22.270767+0000","last_unstale":"2024-03-24T11:40:00.373079+0000","last_undegraded":"2024-03-24T11:40:00.373079+0000","last_fullsized":"2024-03-24T11:40:00.373079+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:21:10.787895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051812070000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375105+0000","last_change":"2024-03-24T11:39:48.132616+0000","last_active":"2024-03-24T11:40:00.375105+0000","last_peered":"2024-03-24T11:40:00.375105+0000","last_clean":"2024-03-24T11:40:00.375105+0000","last_became_active":"2024-03-24T11:39:48.125273+0000","last_became_peered":"2024-03-24T11:39:48.125273+0000","last_unstale":"2024-03-24T11:40:00.375105+0000","last_undegraded":"2024-03-24T11:40:00.375105+0000","last_fullsized":"2024-03-24T11:40:00.375105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:52.137684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374451+0000","last_change":"2024-03-24T11:39:26.343946+0000","last_active":"2024-03-24T11:40:00.374451+0000","last_peered":"2024-03-24T11:40:00.374451+0000","last_clean":"2024-03-24T11:40:00.374451+0000","last_became_active":"2024-03-24T11:39:26.343594+0000","last_became_peered":"2024-03-24T11:39:26.343594+0000","last_unstale":"2024-03-24T11:40:00.374451+0000","last_undegraded":"2024-03-24T11:40:00.374451+0000","last_fullsized":"2024-03-24T11:40:00.374451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:11:53.949036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482610+0000","last_change":"2024-03-24T11:39:28.352754+0000","last_active":"2024-03-24T11:40:00.482610+0000","last_peered":"2024-03-24T11:40:00.482610+0000","last_clean":"2024-03-24T11:40:00.482610+0000","last_became_active":"2024-03-24T11:39:28.352205+0000","last_became_peered":"2024-03-24T11:39:28.352205+0000","last_unstale":"2024-03-24T11:40:00.482610+0000","last_undegraded":"2024-03-24T11:40:00.482610+0000","last_fullsized":"2024-03-24T11:40:00.482610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:13.096353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":276,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:02.481419+0000","last_change":"2024-03-24T11:39:12.747798+0000","last_active":"2024-03-24T11:40:02.481419+0000","last_peered":"2024-03-24T11:40:02.481419+0000","last_clean":"2024-03-24T11:40:02.481419+0000","last_became_active":"2024-03-24T11:39:12.747375+0000","last_became_peered":"2024-03-24T11:39:12.747375+0000","last_unstale":"2024-03-24T11:40:02.481419+0000","last_undegraded":"2024-03-24T11:40:02.481419+0000","last_fullsized":"2024-03-24T11:40:02.481419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:03.562175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:37:03.562175+0000","last_clean_scrub_stamp":"2024-03-24T11:37:03.562175+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:30:00.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376598+0000","last_change":"2024-03-24T11:39:22.272135+0000","last_active":"2024-03-24T11:40:00.376598+0000","last_peered":"2024-03-24T11:40:00.376598+0000","last_clean":"2024-03-24T11:40:00.376598+0000","last_became_active":"2024-03-24T11:39:22.271381+0000","last_became_peered":"2024-03-24T11:39:22.271381+0000","last_unstale":"2024-03-24T11:40:00.376598+0000","last_undegraded":"2024-03-24T11:40:00.376598+0000","last_fullsized":"2024-03-24T11:40:00.376598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:24.922413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":69,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330536+0000","last_change":"2024-03-24T11:39:24.324291+0000","last_active":"2024-03-24T11:40:42.330536+0000","last_peered":"2024-03-24T11:40:42.330536+0000","last_clean":"2024-03-24T11:40:42.330536+0000","last_became_active":"2024-03-24T11:39:24.323949+0000","last_became_peered":"2024-03-24T11:39:24.323949+0000","last_unstale":"2024-03-24T11:40:42.330536+0000","last_undegraded":"2024-03-24T11:40:42.330536+0000","last_fullsized":"2024-03-24T11:40:42.330536+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:47:59.001269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498143+0000","last_change":"2024-03-24T11:39:26.347455+0000","last_active":"2024-03-24T11:40:00.498143+0000","last_peered":"2024-03-24T11:40:00.498143+0000","last_clean":"2024-03-24T11:40:00.498143+0000","last_became_active":"2024-03-24T11:39:26.346834+0000","last_became_peered":"2024-03-24T11:39:26.346834+0000","last_unstale":"2024-03-24T11:40:00.498143+0000","last_undegraded":"2024-03-24T11:40:00.498143+0000","last_fullsized":"2024-03-24T11:40:00.498143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:41:15.692636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214415+0000","last_change":"2024-03-24T11:39:28.360890+0000","last_active":"2024-03-24T11:40:01.214415+0000","last_peered":"2024-03-24T11:40:01.214415+0000","last_clean":"2024-03-24T11:40:01.214415+0000","last_became_active":"2024-03-24T11:39:28.360589+0000","last_became_peered":"2024-03-24T11:39:28.360589+0000","last_unstale":"2024-03-24T11:40:01.214415+0000","last_undegraded":"2024-03-24T11:40:01.214415+0000","last_fullsized":"2024-03-24T11:40:01.214415+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:21.105809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214247+0000","last_change":"2024-03-24T11:39:22.278954+0000","last_active":"2024-03-24T11:40:01.214247+0000","last_peered":"2024-03-24T11:40:01.214247+0000","last_clean":"2024-03-24T11:40:01.214247+0000","last_became_active":"2024-03-24T11:39:22.278635+0000","last_became_peered":"2024-03-24T11:39:22.278635+0000","last_unstale":"2024-03-24T11:40:01.214247+0000","last_undegraded":"2024-03-24T11:40:01.214247+0000","last_fullsized":"2024-03-24T11:40:01.214247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:43:49.065824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":126,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:49.681337+0000","last_change":"2024-03-24T11:39:26.333995+0000","last_active":"2024-03-24T11:40:49.681337+0000","last_peered":"2024-03-24T11:40:49.681337+0000","last_clean":"2024-03-24T11:40:49.681337+0000","last_became_active":"2024-03-24T11:39:22.275391+0000","last_became_peered":"2024-03-24T11:39:22.275391+0000","last_unstale":"2024-03-24T11:40:49.681337+0000","last_undegraded":"2024-03-24T11:40:49.681337+0000","last_fullsized":"2024-03-24T11:40:49.681337+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:12:29.494066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012529869999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215236+0000","last_change":"2024-03-24T11:39:24.330195+0000","last_active":"2024-03-24T11:40:01.215236+0000","last_peered":"2024-03-24T11:40:01.215236+0000","last_clean":"2024-03-24T11:40:01.215236+0000","last_became_active":"2024-03-24T11:39:24.329552+0000","last_became_peered":"2024-03-24T11:39:24.329552+0000","last_unstale":"2024-03-24T11:40:01.215236+0000","last_undegraded":"2024-03-24T11:40:01.215236+0000","last_fullsized":"2024-03-24T11:40:01.215236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:57.235595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482844+0000","last_change":"2024-03-24T11:39:26.345010+0000","last_active":"2024-03-24T11:40:00.482844+0000","last_peered":"2024-03-24T11:40:00.482844+0000","last_clean":"2024-03-24T11:40:00.482844+0000","last_became_active":"2024-03-24T11:39:26.344464+0000","last_became_peered":"2024-03-24T11:39:26.344464+0000","last_unstale":"2024-03-24T11:40:00.482844+0000","last_undegraded":"2024-03-24T11:40:00.482844+0000","last_fullsized":"2024-03-24T11:40:00.482844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:36:38.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.213871+0000","last_change":"2024-03-24T11:39:29.121114+0000","last_active":"2024-03-24T11:40:01.213871+0000","last_peered":"2024-03-24T11:40:01.213871+0000","last_clean":"2024-03-24T11:40:01.213871+0000","last_became_active":"2024-03-24T11:39:29.120705+0000","last_became_peered":"2024-03-24T11:39:29.120705+0000","last_unstale":"2024-03-24T11:40:01.213871+0000","last_undegraded":"2024-03-24T11:40:01.213871+0000","last_fullsized":"2024-03-24T11:40:01.213871+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:34.237225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214646+0000","last_change":"2024-03-24T11:39:22.274676+0000","last_active":"2024-03-24T11:40:01.214646+0000","last_peered":"2024-03-24T11:40:01.214646+0000","last_clean":"2024-03-24T11:40:01.214646+0000","last_became_active":"2024-03-24T11:39:22.273941+0000","last_became_peered":"2024-03-24T11:39:22.273941+0000","last_unstale":"2024-03-24T11:40:01.214646+0000","last_undegraded":"2024-03-24T11:40:01.214646+0000","last_fullsized":"2024-03-24T11:40:01.214646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:33:53.159995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.216164+0000","last_change":"2024-03-24T11:39:24.323375+0000","last_active":"2024-03-24T11:40:01.216164+0000","last_peered":"2024-03-24T11:40:01.216164+0000","last_clean":"2024-03-24T11:40:01.216164+0000","last_became_active":"2024-03-24T11:39:24.323000+0000","last_became_peered":"2024-03-24T11:39:24.323000+0000","last_unstale":"2024-03-24T11:40:01.216164+0000","last_undegraded":"2024-03-24T11:40:01.216164+0000","last_fullsized":"2024-03-24T11:40:01.216164+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:37:42.703654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112774+0000","last_change":"2024-03-24T11:39:26.342326+0000","last_active":"2024-03-24T11:40:01.112774+0000","last_peered":"2024-03-24T11:40:01.112774+0000","last_clean":"2024-03-24T11:40:01.112774+0000","last_became_active":"2024-03-24T11:39:26.341988+0000","last_became_peered":"2024-03-24T11:39:26.341988+0000","last_unstale":"2024-03-24T11:40:01.112774+0000","last_undegraded":"2024-03-24T11:40:01.112774+0000","last_fullsized":"2024-03-24T11:40:01.112774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:40:21.773415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.214110+0000","last_change":"2024-03-24T11:39:28.361252+0000","last_active":"2024-03-24T11:40:01.214110+0000","last_peered":"2024-03-24T11:40:01.214110+0000","last_clean":"2024-03-24T11:40:01.214110+0000","last_became_active":"2024-03-24T11:39:28.360947+0000","last_became_peered":"2024-03-24T11:39:28.360947+0000","last_unstale":"2024-03-24T11:40:01.214110+0000","last_undegraded":"2024-03-24T11:40:01.214110+0000","last_fullsized":"2024-03-24T11:40:01.214110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:00:52.933465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482676+0000","last_change":"2024-03-24T11:39:22.273361+0000","last_active":"2024-03-24T11:40:00.482676+0000","last_peered":"2024-03-24T11:40:00.482676+0000","last_clean":"2024-03-24T11:40:00.482676+0000","last_became_active":"2024-03-24T11:39:22.271889+0000","last_became_peered":"2024-03-24T11:39:22.271889+0000","last_unstale":"2024-03-24T11:40:00.482676+0000","last_undegraded":"2024-03-24T11:40:00.482676+0000","last_fullsized":"2024-03-24T11:40:00.482676+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:48.339910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215143+0000","last_change":"2024-03-24T11:39:24.330329+0000","last_active":"2024-03-24T11:40:01.215143+0000","last_peered":"2024-03-24T11:40:01.215143+0000","last_clean":"2024-03-24T11:40:01.215143+0000","last_became_active":"2024-03-24T11:39:24.329967+0000","last_became_peered":"2024-03-24T11:39:24.329967+0000","last_unstale":"2024-03-24T11:40:01.215143+0000","last_undegraded":"2024-03-24T11:40:01.215143+0000","last_fullsized":"2024-03-24T11:40:01.215143+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:09:05.137006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330363+0000","last_change":"2024-03-24T11:39:26.343215+0000","last_active":"2024-03-24T11:40:42.330363+0000","last_peered":"2024-03-24T11:40:42.330363+0000","last_clean":"2024-03-24T11:40:42.330363+0000","last_became_active":"2024-03-24T11:39:26.342926+0000","last_became_peered":"2024-03-24T11:39:26.342926+0000","last_unstale":"2024-03-24T11:40:42.330363+0000","last_undegraded":"2024-03-24T11:40:42.330363+0000","last_fullsized":"2024-03-24T11:40:42.330363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:27.233167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.498161+0000","last_change":"2024-03-24T11:39:29.121026+0000","last_active":"2024-03-24T11:40:00.498161+0000","last_peered":"2024-03-24T11:40:00.498161+0000","last_clean":"2024-03-24T11:40:00.498161+0000","last_became_active":"2024-03-24T11:39:29.120687+0000","last_became_peered":"2024-03-24T11:39:29.120687+0000","last_unstale":"2024-03-24T11:40:00.498161+0000","last_undegraded":"2024-03-24T11:40:00.498161+0000","last_fullsized":"2024-03-24T11:40:00.498161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:43.442419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'2","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215334+0000","last_change":"2024-03-24T11:39:22.274126+0000","last_active":"2024-03-24T11:40:01.215334+0000","last_peered":"2024-03-24T11:40:01.215334+0000","last_clean":"2024-03-24T11:40:01.215334+0000","last_became_active":"2024-03-24T11:39:22.273797+0000","last_became_peered":"2024-03-24T11:39:22.273797+0000","last_unstale":"2024-03-24T11:40:01.215334+0000","last_undegraded":"2024-03-24T11:40:01.215334+0000","last_fullsized":"2024-03-24T11:40:01.215334+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:03:49.642051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376801+0000","last_change":"2024-03-24T11:39:24.321604+0000","last_active":"2024-03-24T11:40:00.376801+0000","last_peered":"2024-03-24T11:40:00.376801+0000","last_clean":"2024-03-24T11:40:00.376801+0000","last_became_active":"2024-03-24T11:39:24.321287+0000","last_became_peered":"2024-03-24T11:39:24.321287+0000","last_unstale":"2024-03-24T11:40:00.376801+0000","last_undegraded":"2024-03-24T11:40:00.376801+0000","last_fullsized":"2024-03-24T11:40:00.376801+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:39.522487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330437+0000","last_change":"2024-03-24T11:39:26.336388+0000","last_active":"2024-03-24T11:40:42.330437+0000","last_peered":"2024-03-24T11:40:42.330437+0000","last_clean":"2024-03-24T11:40:42.330437+0000","last_became_active":"2024-03-24T11:39:26.336079+0000","last_became_peered":"2024-03-24T11:39:26.336079+0000","last_unstale":"2024-03-24T11:40:42.330437+0000","last_undegraded":"2024-03-24T11:40:42.330437+0000","last_fullsized":"2024-03-24T11:40:42.330437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:52:21.640866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376273+0000","last_change":"2024-03-24T11:39:28.353587+0000","last_active":"2024-03-24T11:40:00.376273+0000","last_peered":"2024-03-24T11:40:00.376273+0000","last_clean":"2024-03-24T11:40:00.376273+0000","last_became_active":"2024-03-24T11:39:28.353261+0000","last_became_peered":"2024-03-24T11:39:28.353261+0000","last_unstale":"2024-03-24T11:40:00.376273+0000","last_undegraded":"2024-03-24T11:40:00.376273+0000","last_fullsized":"2024-03-24T11:40:00.376273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:20:30.018449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376033+0000","last_change":"2024-03-24T11:39:22.272865+0000","last_active":"2024-03-24T11:40:00.376033+0000","last_peered":"2024-03-24T11:40:00.376033+0000","last_clean":"2024-03-24T11:40:00.376033+0000","last_became_active":"2024-03-24T11:39:22.272573+0000","last_became_peered":"2024-03-24T11:39:22.272573+0000","last_unstale":"2024-03-24T11:40:00.376033+0000","last_undegraded":"2024-03-24T11:40:00.376033+0000","last_fullsized":"2024-03-24T11:40:00.376033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:37.543022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374687+0000","last_change":"2024-03-24T11:39:24.323346+0000","last_active":"2024-03-24T11:40:00.374687+0000","last_peered":"2024-03-24T11:40:00.374687+0000","last_clean":"2024-03-24T11:40:00.374687+0000","last_became_active":"2024-03-24T11:39:24.323056+0000","last_became_peered":"2024-03-24T11:39:24.323056+0000","last_unstale":"2024-03-24T11:40:00.374687+0000","last_undegraded":"2024-03-24T11:40:00.374687+0000","last_fullsized":"2024-03-24T11:40:00.374687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:47.808051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376610+0000","last_change":"2024-03-24T11:39:26.340801+0000","last_active":"2024-03-24T11:40:00.376610+0000","last_peered":"2024-03-24T11:40:00.376610+0000","last_clean":"2024-03-24T11:40:00.376610+0000","last_became_active":"2024-03-24T11:39:26.340436+0000","last_became_peered":"2024-03-24T11:39:26.340436+0000","last_unstale":"2024-03-24T11:40:00.376610+0000","last_undegraded":"2024-03-24T11:40:00.376610+0000","last_fullsized":"2024-03-24T11:40:00.376610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:44.979604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482425+0000","last_change":"2024-03-24T11:39:28.356405+0000","last_active":"2024-03-24T11:40:00.482425+0000","last_peered":"2024-03-24T11:40:00.482425+0000","last_clean":"2024-03-24T11:40:00.482425+0000","last_became_active":"2024-03-24T11:39:28.356073+0000","last_became_peered":"2024-03-24T11:39:28.356073+0000","last_unstale":"2024-03-24T11:40:00.482425+0000","last_undegraded":"2024-03-24T11:40:00.482425+0000","last_fullsized":"2024-03-24T11:40:00.482425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:21:52.998338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330527+0000","last_change":"2024-03-24T11:39:26.342944+0000","last_active":"2024-03-24T11:40:42.330527+0000","last_peered":"2024-03-24T11:40:42.330527+0000","last_clean":"2024-03-24T11:40:42.330527+0000","last_became_active":"2024-03-24T11:39:26.340796+0000","last_became_peered":"2024-03-24T11:39:26.340796+0000","last_unstale":"2024-03-24T11:40:42.330527+0000","last_undegraded":"2024-03-24T11:40:42.330527+0000","last_fullsized":"2024-03-24T11:40:42.330527+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:38:35.159323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373582+0000","last_change":"2024-03-24T11:39:22.270253+0000","last_active":"2024-03-24T11:40:00.373582+0000","last_peered":"2024-03-24T11:40:00.373582+0000","last_clean":"2024-03-24T11:40:00.373582+0000","last_became_active":"2024-03-24T11:39:22.269983+0000","last_became_peered":"2024-03-24T11:39:22.269983+0000","last_unstale":"2024-03-24T11:40:00.373582+0000","last_undegraded":"2024-03-24T11:40:00.373582+0000","last_fullsized":"2024-03-24T11:40:00.373582+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:08:57.543296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":29,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.374871+0000","last_change":"2024-03-24T11:39:48.140282+0000","last_active":"2024-03-24T11:40:00.374871+0000","last_peered":"2024-03-24T11:40:00.374871+0000","last_clean":"2024-03-24T11:40:00.374871+0000","last_became_active":"2024-03-24T11:39:48.126374+0000","last_became_peered":"2024-03-24T11:39:48.126374+0000","last_unstale":"2024-03-24T11:40:00.374871+0000","last_undegraded":"2024-03-24T11:40:00.374871+0000","last_fullsized":"2024-03-24T11:40:00.374871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:05:42.578844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373834+0000","last_change":"2024-03-24T11:39:29.121297+0000","last_active":"2024-03-24T11:40:00.373834+0000","last_peered":"2024-03-24T11:40:00.373834+0000","last_clean":"2024-03-24T11:40:00.373834+0000","last_became_active":"2024-03-24T11:39:29.120896+0000","last_became_peered":"2024-03-24T11:39:29.120896+0000","last_unstale":"2024-03-24T11:40:00.373834+0000","last_undegraded":"2024-03-24T11:40:00.373834+0000","last_fullsized":"2024-03-24T11:40:00.373834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:54.731060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.482776+0000","last_change":"2024-03-24T11:39:26.344752+0000","last_active":"2024-03-24T11:40:00.482776+0000","last_peered":"2024-03-24T11:40:00.482776+0000","last_clean":"2024-03-24T11:40:00.482776+0000","last_became_active":"2024-03-24T11:39:26.344019+0000","last_became_peered":"2024-03-24T11:39:26.344019+0000","last_unstale":"2024-03-24T11:40:00.482776+0000","last_undegraded":"2024-03-24T11:40:00.482776+0000","last_fullsized":"2024-03-24T11:40:00.482776+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:13:36.517702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.376696+0000","last_change":"2024-03-24T11:39:22.271959+0000","last_active":"2024-03-24T11:40:00.376696+0000","last_peered":"2024-03-24T11:40:00.376696+0000","last_clean":"2024-03-24T11:40:00.376696+0000","last_became_active":"2024-03-24T11:39:22.271671+0000","last_became_peered":"2024-03-24T11:39:22.271671+0000","last_unstale":"2024-03-24T11:40:00.376696+0000","last_undegraded":"2024-03-24T11:40:00.376696+0000","last_fullsized":"2024-03-24T11:40:00.376696+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:48:00.296407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377792+0000","last_change":"2024-03-24T11:39:24.326366+0000","last_active":"2024-03-24T11:40:00.377792+0000","last_peered":"2024-03-24T11:40:00.377792+0000","last_clean":"2024-03-24T11:40:00.377792+0000","last_became_active":"2024-03-24T11:39:24.325774+0000","last_became_peered":"2024-03-24T11:39:24.325774+0000","last_unstale":"2024-03-24T11:40:00.377792+0000","last_undegraded":"2024-03-24T11:40:00.377792+0000","last_fullsized":"2024-03-24T11:40:00.377792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:48.678848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.375019+0000","last_change":"2024-03-24T11:39:28.354215+0000","last_active":"2024-03-24T11:40:00.375019+0000","last_peered":"2024-03-24T11:40:00.375019+0000","last_clean":"2024-03-24T11:40:00.375019+0000","last_became_active":"2024-03-24T11:39:28.353929+0000","last_became_peered":"2024-03-24T11:39:28.353929+0000","last_unstale":"2024-03-24T11:40:00.375019+0000","last_undegraded":"2024-03-24T11:40:00.375019+0000","last_fullsized":"2024-03-24T11:40:00.375019+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:28:01.666879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.215005+0000","last_change":"2024-03-24T11:39:26.342730+0000","last_active":"2024-03-24T11:40:01.215005+0000","last_peered":"2024-03-24T11:40:01.215005+0000","last_clean":"2024-03-24T11:40:01.215005+0000","last_became_active":"2024-03-24T11:39:26.341904+0000","last_became_peered":"2024-03-24T11:39:26.341904+0000","last_unstale":"2024-03-24T11:40:01.215005+0000","last_undegraded":"2024-03-24T11:40:01.215005+0000","last_fullsized":"2024-03-24T11:40:01.215005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:34:17.492295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:01.112688+0000","last_change":"2024-03-24T11:39:22.274718+0000","last_active":"2024-03-24T11:40:01.112688+0000","last_peered":"2024-03-24T11:40:01.112688+0000","last_clean":"2024-03-24T11:40:01.112688+0000","last_became_active":"2024-03-24T11:39:22.273663+0000","last_became_peered":"2024-03-24T11:39:22.273663+0000","last_unstale":"2024-03-24T11:40:01.112688+0000","last_undegraded":"2024-03-24T11:40:01.112688+0000","last_fullsized":"2024-03-24T11:40:01.112688+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:37.272429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.377372+0000","last_change":"2024-03-24T11:39:24.327870+0000","last_active":"2024-03-24T11:40:00.377372+0000","last_peered":"2024-03-24T11:40:00.377372+0000","last_clean":"2024-03-24T11:40:00.377372+0000","last_became_active":"2024-03-24T11:39:24.327329+0000","last_became_peered":"2024-03-24T11:39:24.327329+0000","last_unstale":"2024-03-24T11:40:00.377372+0000","last_undegraded":"2024-03-24T11:40:00.377372+0000","last_fullsized":"2024-03-24T11:40:00.377372+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:18:24.755060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378136+0000","last_change":"2024-03-24T11:39:28.350475+0000","last_active":"2024-03-24T11:40:00.378136+0000","last_peered":"2024-03-24T11:40:00.378136+0000","last_clean":"2024-03-24T11:40:00.378136+0000","last_became_active":"2024-03-24T11:39:28.350208+0000","last_became_peered":"2024-03-24T11:39:28.350208+0000","last_unstale":"2024-03-24T11:40:00.378136+0000","last_undegraded":"2024-03-24T11:40:00.378136+0000","last_fullsized":"2024-03-24T11:40:00.378136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:27.322185+0000","last_clean_scrub_stamp":"2024-03-24T11:39:27.322185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:24:20.858087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:42.330363+0000","last_change":"2024-03-24T11:39:26.344831+0000","last_active":"2024-03-24T11:40:42.330363+0000","last_peered":"2024-03-24T11:40:42.330363+0000","last_clean":"2024-03-24T11:40:42.330363+0000","last_became_active":"2024-03-24T11:39:26.344445+0000","last_became_peered":"2024-03-24T11:39:26.344445+0000","last_unstale":"2024-03-24T11:40:42.330363+0000","last_undegraded":"2024-03-24T11:40:42.330363+0000","last_fullsized":"2024-03-24T11:40:42.330363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:25.308617+0000","last_clean_scrub_stamp":"2024-03-24T11:39:25.308617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:11:52.503048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.373673+0000","last_change":"2024-03-24T11:39:22.271027+0000","last_active":"2024-03-24T11:40:00.373673+0000","last_peered":"2024-03-24T11:40:00.373673+0000","last_clean":"2024-03-24T11:40:00.373673+0000","last_became_active":"2024-03-24T11:39:22.270729+0000","last_became_peered":"2024-03-24T11:39:22.270729+0000","last_unstale":"2024-03-24T11:40:00.373673+0000","last_undegraded":"2024-03-24T11:40:00.373673+0000","last_fullsized":"2024-03-24T11:40:00.373673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:21.253281+0000","last_clean_scrub_stamp":"2024-03-24T11:39:21.253281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:25:54.270435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-24T11:40:00.378060+0000","last_change":"2024-03-24T11:39:24.323492+0000","last_active":"2024-03-24T11:40:00.378060+0000","last_peered":"2024-03-24T11:40:00.378060+0000","last_clean":"2024-03-24T11:40:00.378060+0000","last_became_active":"2024-03-24T11:39:24.323175+0000","last_became_peered":"2024-03-24T11:39:24.323175+0000","last_unstale":"2024-03-24T11:40:00.378060+0000","last_undegraded":"2024-03-24T11:40:00.378060+0000","last_fullsized":"2024-03-24T11:40:00.378060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:23.298385+0000","last_clean_scrub_stamp":"2024-03-24T11:39:23.298385+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:31.912447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201302,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1310720,"data_stored":869990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462940,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1286144,"data_stored":868925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":745472,"data_stored":281065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1314816,"data_stored":870117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":716800,"data_stored":280110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":279448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":729088,"data_stored":279573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724577,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":700416,"data_stored":279758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:40:56.919 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:40:56.919 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:40:56.920 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:40:56.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:40:57.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:57.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: from='client.34631 v1:172.21.15.114:0/1997801331' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:40:57.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:56 smithi119 ceph-mon[33714]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: from='client.34631 v1:172.21.15.114:0/1997801331' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:40:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[42466]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: from='client.34631 v1:172.21.15.114:0/1997801331' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:40:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:56 smithi114 ceph-mon[30312]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:57.789 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:59.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.644+0000 7f2af4e72700 1 -- 172.21.15.114:0/2999167432 >> v1:172.21.15.114:6790/0 conn(0x7f2af0107a20 legacy=0x7f2af0107e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:59.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:40:59 smithi114 ceph-mon[30312]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:59.645 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:40:59 smithi114 ceph-mon[42466]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.644+0000 7f2af4e72700 1 -- 172.21.15.114:0/2999167432 shutdown_connections 2024-03-24T11:40:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.644+0000 7f2af4e72700 1 -- 172.21.15.114:0/2999167432 >> 172.21.15.114:0/2999167432 conn(0x7f2af00fd550 msgr2=0x7f2af00ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.644+0000 7f2af4e72700 1 -- 172.21.15.114:0/2999167432 shutdown_connections 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.644+0000 7f2af4e72700 1 -- 172.21.15.114:0/2999167432 wait complete. 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.645+0000 7f2af4e72700 1 Processor -- start 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.645+0000 7f2af4e72700 1 -- start start 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2af4e72700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2af01a68e0 con 0x7f2af010a8d0 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2af4e72700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2af01a7ac0 con 0x7f2af010e4a0 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2af4e72700 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2af01a8cc0 con 0x7f2af0107a20 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2aee59c700 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2af0107a20 0x7f2af010db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39404/0 (socket says 172.21.15.114:39404) 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2aee59c700 1 -- 172.21.15.114:0/2051958236 learned_addr learned my addr 172.21.15.114:0/2051958236 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861191847 0 0) 0x7f2af01a68e0 con 0x7f2af010a8d0 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad80035b0 con 0x7f2af010a8d0 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214825328 0 0) 0x7f2af01a8cc0 con 0x7f2af0107a20 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af01a68e0 con 0x7f2af0107a20 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645044650 0 0) 0x7f2af01a7ac0 con 0x7f2af010e4a0 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af01a8cc0 con 0x7f2af010e4a0 2024-03-24T11:40:59.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303050935 0 0) 0x7f2ad80035b0 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af01a7ac0 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2ae4001bd0 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675879286 0 0) 0x7f2af01a68e0 con 0x7f2af0107a20 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad80035b0 con 0x7f2af0107a20 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2ae80015d0 con 0x7f2af0107a20 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2419933314 0 0) 0x7f2af01a8cc0 con 0x7f2af010e4a0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af01a68e0 con 0x7f2af010e4a0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2ae0001770 con 0x7f2af010e4a0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 326220424 0 0) 0x7f2af01a7ac0 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 >> v1:172.21.15.114:6790/0 conn(0x7f2af0107a20 legacy=0x7f2af010db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 >> v1:172.21.15.119:6789/0 conn(0x7f2af010e4a0 legacy=0x7f2af01a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.646+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af01a9ec0 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.647+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af01a8eb0 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.647+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af01a9330 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.647+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2ae4003e90 con 0x7f2af010a8d0 2024-03-24T11:40:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.647+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2ae4004ce0 con 0x7f2af010a8d0 2024-03-24T11:40:59.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.647+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad0005120 con 0x7f2af010a8d0 2024-03-24T11:40:59.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.648+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f2ae401b230 con 0x7f2af010a8d0 2024-03-24T11:40:59.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.648+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (unknown 880917791 0 0) 0x7f2ae40873e0 con 0x7f2af010a8d0 2024-03-24T11:40:59.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.653+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f2ae40548c0 con 0x7f2af010a8d0 2024-03-24T11:40:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:40:59 smithi119 ceph-mon[33714]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:59.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.915+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2ad0005950 con 0x7f2af010a8d0 2024-03-24T11:40:59.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.915+0000 7f2adf7fe700 1 -- 172.21.15.114:0/2051958236 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f2ae4052c30 con 0x7f2af010a8d0 2024-03-24T11:40:59.917 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-24T11:40:59.917 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:40:59.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.918+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 >> v1:172.21.15.114:6800/1253397276 conn(0x7f2ad806e590 legacy=0x7f2ad8070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:59.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.918+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 >> v1:172.21.15.114:6789/0 conn(0x7f2af010a8d0 legacy=0x7f2af01a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:40:59.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.918+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 shutdown_connections 2024-03-24T11:40:59.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.918+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 >> 172.21.15.114:0/2051958236 conn(0x7f2af00fd550 msgr2=0x7f2af0101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:40:59.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.919+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 shutdown_connections 2024-03-24T11:40:59.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-24T11:40:59.919+0000 7f2af4e72700 1 -- 172.21.15.114:0/2051958236 wait complete. 2024-03-24T11:41:00.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2051958236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:41:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2051958236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:41:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2051958236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:41:00.926 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:41:00.926 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:41:00.927 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T11:41:00.940 INFO:tasks.workunit:Pulling workunits from ref f73f1dfd96849300c4322bd7259a3ab37f94c200 2024-03-24T11:41:00.940 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T11:41:00.941 DEBUG:teuthology.orchestra.run.smithi114:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T11:41:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:41:00.957 INFO:teuthology.orchestra.run.smithi114.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-24T11:41:00.958 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-24T11:41:01.012 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-24T11:41:01.012 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-24T11:41:01.067 INFO:tasks.workunit:timeout=3h 2024-03-24T11:41:01.067 INFO:tasks.workunit:cleanup=True 2024-03-24T11:41:01.067 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f73f1dfd96849300c4322bd7259a3ab37f94c200 2024-03-24T11:41:01.123 INFO:tasks.workunit.client.0.smithi114.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T11:41:01.630 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[30312]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.630 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.630 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:01.630 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.630 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.631 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[42466]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.631 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.631 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:01.631 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.631 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:41:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:41:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:01 smithi119 ceph-mon[33714]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:01.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:01.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:02.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:02.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:02.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:02 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:03 smithi114 ceph-mon[42466]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:41:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:03 smithi114 ceph-mon[42466]: Reconfiguring daemon alertmanager.a on smithi114 2024-03-24T11:41:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:03 smithi114 ceph-mon[42466]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:03 smithi114 ceph-mon[30312]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:41:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:03 smithi114 ceph-mon[30312]: Reconfiguring daemon alertmanager.a on smithi114 2024-03-24T11:41:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:03 smithi114 ceph-mon[30312]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:03 smithi119 ceph-mon[33714]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:41:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:03 smithi119 ceph-mon[33714]: Reconfiguring daemon alertmanager.a on smithi114 2024-03-24T11:41:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:03 smithi119 ceph-mon[33714]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:04.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:04 smithi114 ceph-mon[30312]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:04.725 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:04 smithi114 ceph-mon[42466]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:04.725 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:41:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:04 smithi119 ceph-mon[33714]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:05.989 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:05 smithi114 systemd[1]: Stopping Ceph alertmanager.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:41:06.489 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:06 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[78115]: ts=2024-03-24T11:41:06.230Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:41:06.489 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:06 smithi114 podman[81383]: 2024-03-24 11:41:06.251055315 +0000 UTC m=+0.333347236 container died b46fb4351c0a063f27bb81d8042ec68eb2a45b0c797539f3e21278010a46facf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:06.962 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:06 smithi114 podman[81383]: 2024-03-24 11:41:06.686538802 +0000 UTC m=+0.768830732 container cleanup b46fb4351c0a063f27bb81d8042ec68eb2a45b0c797539f3e21278010a46facf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:06.963 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:06 smithi114 bash[81383]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:41:07.238 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:06 smithi114 podman[81402]: 2024-03-24 11:41:06.962989146 +0000 UTC m=+0.715238216 container remove b46fb4351c0a063f27bb81d8042ec68eb2a45b0c797539f3e21278010a46facf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:07.238 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 podman[81402]: 2024-03-24 11:41:07.063615438 +0000 UTC m=+0.815864508 volume remove 2e271f595fa6053e066939628b4757f5a11f3af59b21bcf4c657beedb888e5a3 2024-03-24T11:41:07.573 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-mon[30312]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:07.573 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-mon[42466]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:07.573 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:41:07.573 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 systemd[1]: Stopped Ceph alertmanager.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:41:07.574 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 systemd[1]: Starting Ceph alertmanager.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:41:07.574 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 podman[81503]: 2024-03-24 11:41:07.464573308 +0000 UTC m=+0.111714500 volume create d659fe8b7130334d54b9c67148bfe2360ee02078e7339d477f093875ffe04ec6 2024-03-24T11:41:07.574 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 podman[81503]: 2024-03-24 11:41:07.371589702 +0000 UTC m=+0.018730909 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-24T11:41:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:07 smithi119 ceph-mon[33714]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:07.876 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 podman[81503]: 2024-03-24 11:41:07.57346363 +0000 UTC m=+0.220604824 container create 64dc685b8a6da1024b22b31701928592f12e3932a56608b37732f988b75083ae (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:08.239 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 podman[81503]: 2024-03-24 11:41:07.876572733 +0000 UTC m=+0.523713936 container init 64dc685b8a6da1024b22b31701928592f12e3932a56608b37732f988b75083ae (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:08.239 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 podman[81503]: 2024-03-24 11:41:07.881700482 +0000 UTC m=+0.528841684 container start 64dc685b8a6da1024b22b31701928592f12e3932a56608b37732f988b75083ae (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:08.239 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.893Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:41:08.239 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.893Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:41:08.239 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.894Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.114 port=9094 2024-03-24T11:41:08.239 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.895Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:41:08.240 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.917Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:41:08.240 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.918Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:41:08.240 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.921Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:41:08.240 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:07.921Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:41:08.240 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 bash[81503]: 64dc685b8a6da1024b22b31701928592f12e3932a56608b37732f988b75083ae 2024-03-24T11:41:08.240 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:07 smithi114 systemd[1]: Started Ceph alertmanager.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:41:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:09 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:09 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:09 smithi119 ceph-mon[33714]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:41:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:09 smithi119 ceph-mon[33714]: Reconfiguring daemon prometheus.a on smithi119 2024-03-24T11:41:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:09 smithi119 ceph-mon[33714]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[42466]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[42466]: Reconfiguring daemon prometheus.a on smithi119 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[42466]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[30312]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:41:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[30312]: Reconfiguring daemon prometheus.a on smithi119 2024-03-24T11:41:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-mon[30312]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:10.239 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:09 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:09.896Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000262935s 2024-03-24T11:41:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:11 smithi114 ceph-mon[42466]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:11 smithi114 ceph-mon[30312]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:11 smithi119 ceph-mon[33714]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:11.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:41:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:41:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:12 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:12 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:12.746 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:12 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:13.718 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:13 smithi119 ceph-mon[33714]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:13 smithi114 ceph-mon[42466]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:13.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:13 smithi114 ceph-mon[30312]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:14 smithi119 ceph-mon[33714]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:14.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:14 smithi114 ceph-mon[30312]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:14.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:14 smithi114 ceph-mon[42466]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:14.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:41:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:15.304 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 systemd[1]: Stopping Ceph prometheus.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:41:15.647 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:15.647 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:41:15.647 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.339Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.340Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.340Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[64193]: ts=2024-03-24T11:41:15.340Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T11:41:15.648 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 podman[67695]: 2024-03-24 11:41:15.349993118 +0000 UTC m=+0.138064369 container died e8f9fae1104f9ff133d0a9e9f3b59ebc5071ee15fea319c05a39fb05f7ce2eb9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:15.987 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 podman[67695]: 2024-03-24 11:41:15.753494283 +0000 UTC m=+0.541565528 container cleanup e8f9fae1104f9ff133d0a9e9f3b59ebc5071ee15fea319c05a39fb05f7ce2eb9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:15.988 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 bash[67695]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:41:15.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:16.304 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:15 smithi119 podman[67714]: 2024-03-24 11:41:15.988118508 +0000 UTC m=+0.639090459 container remove e8f9fae1104f9ff133d0a9e9f3b59ebc5071ee15fea319c05a39fb05f7ce2eb9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:16.304 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T11:41:16.304 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 systemd[1]: Stopped Ceph prometheus.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:41:16.304 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.112s CPU time. 2024-03-24T11:41:16.304 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 systemd[1]: Starting Ceph prometheus.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T11:41:16.593 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 podman[67815]: 2024-03-24 11:41:16.225114601 +0000 UTC m=+0.018893421 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T11:41:16.593 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 podman[67815]: 2024-03-24 11:41:16.355912321 +0000 UTC m=+0.149691175 container create e0c807da30bec18da9f5559849e0267746eb32b512233ababc38c9405c103c76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:16.868 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-mon[30312]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:16.868 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-mon[42466]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:16.868 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STOPPING 2024-03-24T11:41:16.868 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:41:16.868 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STOPPED 2024-03-24T11:41:16.869 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STARTING 2024-03-24T11:41:16.869 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Serving on http://:::9283 2024-03-24T11:41:16.869 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STARTED 2024-03-24T11:41:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-mon[33714]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:17.055 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 podman[67815]: 2024-03-24 11:41:16.608190317 +0000 UTC m=+0.401969138 container init e0c807da30bec18da9f5559849e0267746eb32b512233ababc38c9405c103c76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:17.055 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 podman[67815]: 2024-03-24 11:41:16.61330843 +0000 UTC m=+0.407087242 container start e0c807da30bec18da9f5559849e0267746eb32b512233ababc38c9405c103c76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:17.055 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.637Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:41:17.055 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.637Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:41:17.056 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.637Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi119 (none))" 2024-03-24T11:41:17.056 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.637Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:41:17.056 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.637Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:41:17.056 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.639Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:41:17.056 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.639Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:41:17.056 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.640Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:41:17.057 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.640Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:41:17.057 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.642Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:41:17.057 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.643Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=8.602µs 2024-03-24T11:41:17.057 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.643Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:41:17.057 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-24T11:41:17.057 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.643Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.108µs wal_replay_duration=438.602µs wbl_replay_duration=97ns total_replay_duration=494.899µs 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.644Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.644Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.644Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.662Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.880132ms db_storage=760ns remote_storage=1.046µs web_handler=381ns query_engine=542ns scrape=377.654µs scrape_sd=90.969µs notify=17.352µs notify_sd=18.388µs rules=17.178644ms tracing=5.109µs 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.662Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T11:41:16.662Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:41:17.058 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 bash[67815]: e0c807da30bec18da9f5559849e0267746eb32b512233ababc38c9405c103c76 2024-03-24T11:41:17.059 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 11:41:16 smithi119 systemd[1]: Started Ceph prometheus.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STOPPING 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STOPPED 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STARTING 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Serving on http://:::9283 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STARTED 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STOPPING 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STOPPED 2024-03-24T11:41:17.239 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:16 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:16] ENGINE Bus STARTING 2024-03-24T11:41:17.240 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:17] ENGINE Serving on http://:::9283 2024-03-24T11:41:17.240 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:11:41:17] ENGINE Bus STARTED 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi119.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:41:17.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi119.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi119.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi119.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:41:17.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi119.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi119.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi119.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi119.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:17.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi119.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi119.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi119.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mon.0 v1:172.21.15.114:0/3810949987' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi119.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:18.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:17 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:18.238 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 11:41:17 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T11:41:17.898Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002657645s 2024-03-24T11:41:18.789 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:18 smithi119 ceph-mon[33714]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:18 smithi114 ceph-mon[30312]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:18.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:18 smithi114 ceph-mon[42466]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:20 smithi114 ceph-mon[30312]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:21.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:21.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:21.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:20 smithi114 ceph-mon[42466]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:20 smithi119 ceph-mon[33714]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:21.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:41:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:41:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:22 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:22 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:22 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:22 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:22 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:22.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:22.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:22.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:22 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:41:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:23 smithi119 ceph-mon[33714]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:23 smithi114 ceph-mon[42466]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:23 smithi114 ceph-mon[30312]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:41:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:25 smithi114 ceph-mon[30312]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:25 smithi114 ceph-mon[42466]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:25 smithi119 ceph-mon[33714]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:27 smithi114 ceph-mon[30312]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:27 smithi114 ceph-mon[42466]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:27 smithi119 ceph-mon[33714]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:28 smithi119 ceph-mon[33714]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:28.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:28 smithi114 ceph-mon[30312]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:28.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:28 smithi114 ceph-mon[42466]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:31.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:31 smithi114 ceph-mon[42466]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:31.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:31.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:31 smithi114 ceph-mon[30312]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:31.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:31.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:41:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:41:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:31 smithi119 ceph-mon[33714]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:33.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:33.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:33 smithi114 ceph-mon[30312]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:33 smithi114 ceph-mon[42466]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:33 smithi119 ceph-mon[33714]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:34 smithi119 ceph-mon[33714]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:34 smithi114 ceph-mon[30312]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:34 smithi114 ceph-mon[42466]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:34.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:41:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:37 smithi114 ceph-mon[30312]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:37 smithi114 ceph-mon[42466]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:37 smithi119 ceph-mon[33714]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:39.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:39 smithi114 ceph-mon[30312]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:39.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:39 smithi114 ceph-mon[42466]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:39 smithi119 ceph-mon[33714]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:40.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:40 smithi114 ceph-mon[30312]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:40 smithi114 ceph-mon[42466]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:40 smithi119 ceph-mon[33714]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:41.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:41:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:41:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:43 smithi114 ceph-mon[30312]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:43.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:43 smithi114 ceph-mon[42466]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.765 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:43.765 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:43 smithi119 ceph-mon[33714]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:44 smithi119 ceph-mon[33714]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:44 smithi114 ceph-mon[30312]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:44 smithi114 ceph-mon[42466]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:44.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:41:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:45.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:45.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:46.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:46.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:46 smithi114 ceph-mon[30312]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:46.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:46 smithi114 ceph-mon[42466]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:46 smithi119 ceph-mon[33714]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:49 smithi114 ceph-mon[30312]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:49 smithi114 ceph-mon[42466]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:49 smithi119 ceph-mon[33714]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:50 smithi114 ceph-mon[30312]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:50 smithi114 ceph-mon[42466]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:50 smithi119 ceph-mon[33714]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:51.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:41:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:41:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:53 smithi114 ceph-mon[30312]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:53 smithi114 ceph-mon[42466]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:53 smithi119 ceph-mon[33714]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:54.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:54 smithi114 ceph-mon[30312]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:54.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:54 smithi114 ceph-mon[42466]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:41:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:41:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:55.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:54 smithi119 ceph-mon[33714]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:56.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:56 smithi119 ceph-mon[33714]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:56.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:56 smithi114 ceph-mon[30312]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:56 smithi114 ceph-mon[42466]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:41:58 smithi119 ceph-mon[33714]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:58.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:41:58 smithi114 ceph-mon[30312]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:58.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:41:58 smithi114 ceph-mon[42466]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:00 smithi119 ceph-mon[33714]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:00.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:00 smithi114 ceph-mon[30312]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:00.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:00 smithi114 ceph-mon[42466]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:01.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:42:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:42:01.889 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:01.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:02 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:02 smithi119 ceph-mon[33714]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:02 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:02 smithi114 ceph-mon[30312]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:02.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:02 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:02.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:02 smithi114 ceph-mon[42466]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:04 smithi119 ceph-mon[33714]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:04.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:04 smithi114 ceph-mon[30312]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:04.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:04 smithi114 ceph-mon[42466]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:04.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:42:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:42:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:06.574 INFO:tasks.workunit.client.0.smithi114.stderr:Note: switching to 'f73f1dfd96849300c4322bd7259a3ab37f94c200'. 2024-03-24T11:42:06.574 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-03-24T11:42:06.574 INFO:tasks.workunit.client.0.smithi114.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T11:42:06.574 INFO:tasks.workunit.client.0.smithi114.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T11:42:06.574 INFO:tasks.workunit.client.0.smithi114.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T11:42:06.574 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-03-24T11:42:06.574 INFO:tasks.workunit.client.0.smithi114.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr: git switch -c 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr:Or undo this operation with: 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr: git switch - 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-03-24T11:42:06.575 INFO:tasks.workunit.client.0.smithi114.stderr:HEAD is now at f73f1dfd968 Merge pull request #56388 from rhcs-dashboard/wip-64960-squid 2024-03-24T11:42:06.578 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T11:42:06.611 INFO:tasks.workunit.client.0.smithi114.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T11:42:06.613 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T11:42:06.613 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T11:42:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:06 smithi119 ceph-mon[33714]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:06 smithi114 ceph-mon[30312]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:06.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:06 smithi114 ceph-mon[42466]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:07.502 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T11:42:07.588 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T11:42:07.629 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T11:42:07.630 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T11:42:07.630 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T11:42:07.670 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T11:42:07.673 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-24T11:42:07.673 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T11:42:07.729 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-24T11:42:07.730 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-24T11:42:07.730 DEBUG:teuthology.orchestra.run.smithi114:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f73f1dfd96849300c4322bd7259a3ab37f94c200 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ parallel=1 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' '' = --serial ']' 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ crimson=0 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' '' = --crimson ']' 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ color= 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -t 1 ']' 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-24T11:42:07.798 INFO:tasks.workunit.client.0.smithi114.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-24T11:42:07.799 INFO:tasks.workunit.client.0.smithi114.stderr:+ declare -A pids 2024-03-24T11:42:07.799 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.799 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.799 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_aio 2024-03-24T11:42:07.800 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_aio' 2024-03-24T11:42:07.800 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_aio 2024-03-24T11:42:07.800 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.800 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_aio 2024-03-24T11:42:07.800 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82340 2024-03-24T11:42:07.800 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_aio on pid 82340' 2024-03-24T11:42:07.801 INFO:tasks.workunit.client.0.smithi114.stdout:test api_aio on pid 82340 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82340 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_aio_pp 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_aio_pp' 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_aio_pp 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.804 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.805 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_aio_pp 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82347 2024-03-24T11:42:07.806 INFO:tasks.workunit.client.0.smithi114.stdout:test api_aio_pp on pid 82347 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_aio_pp on pid 82347' 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82347 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_io 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_io' 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_io 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.811 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.812 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.813 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_io 2024-03-24T11:42:07.814 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82357 2024-03-24T11:42:07.815 INFO:tasks.workunit.client.0.smithi114.stdout:test api_io on pid 82357 2024-03-24T11:42:07.819 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_io on pid 82357' 2024-03-24T11:42:07.819 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82357 2024-03-24T11:42:07.819 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.819 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.819 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_io_pp 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_io_pp' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_io_pp 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.820 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.821 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.822 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.823 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_io_pp 2024-03-24T11:42:07.824 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82368 2024-03-24T11:42:07.824 INFO:tasks.workunit.client.0.smithi114.stdout:test api_io_pp on pid 82368 2024-03-24T11:42:07.829 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_io_pp on pid 82368' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82368 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_asio 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_asio' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.830 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_asio 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.831 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.832 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.833 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.834 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_asio 2024-03-24T11:42:07.835 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82382 2024-03-24T11:42:07.836 INFO:tasks.workunit.client.0.smithi114.stdout:test api_asio on pid 82382 2024-03-24T11:42:07.842 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_asio on pid 82382' 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82382 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_list 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_list' 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_list 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.843 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.844 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82340/exe 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.845 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.846 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.847 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.848 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_list 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82398 2024-03-24T11:42:07.849 INFO:tasks.workunit.client.0.smithi114.stdout:test api_list on pid 82398 2024-03-24T11:42:07.856 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_list on pid 82398' 2024-03-24T11:42:07.856 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82398 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_lock 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_lock' 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.857 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.858 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_aio: /' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.859 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.860 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.861 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.862 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_lock 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.863 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.864 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.864 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.864 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.864 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.865 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.865 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_lock 2024-03-24T11:42:07.865 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.865 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.865 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82417 2024-03-24T11:42:07.865 INFO:tasks.workunit.client.0.smithi114.stdout:test api_lock on pid 82417 2024-03-24T11:42:07.867 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_lock on pid 82417' 2024-03-24T11:42:07.867 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82417 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.868 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_lock_pp 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_lock_pp' 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82347/exe 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.869 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_lock_pp 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.870 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.871 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.872 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.873 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.874 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.875 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82357/exe 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-24T11:42:07.876 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.877 INFO:tasks.workunit.client.0.smithi114.stdout:test api_lock_pp on pid 82439 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_lock_pp 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82439 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_lock_pp on pid 82439' 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82439 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.880 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.881 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_misc 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_misc' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.882 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.883 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.884 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.885 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.886 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_io: /' 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82368/exe 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_io.log 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.887 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.888 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.889 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_misc 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.890 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.891 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.892 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.892 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-24T11:42:07.892 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_misc 2024-03-24T11:42:07.892 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-24T11:42:07.892 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82465 2024-03-24T11:42:07.892 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_misc on pid 82465' 2024-03-24T11:42:07.892 INFO:tasks.workunit.client.0.smithi114.stdout:test api_misc on pid 82465 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82465 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_misc_pp 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82382/exe 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_misc_pp' 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.896 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.897 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.898 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.899 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_misc_pp 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.900 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.901 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_asio: /' 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-24T11:42:07.902 INFO:tasks.workunit.client.0.smithi114.stdout:test api_misc_pp on pid 82484 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.909 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_misc_pp 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.910 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82484 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_misc_pp on pid 82484' 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82484 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_tier_pp 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_tier_pp' 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.911 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.912 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_tier_pp 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82398/exe 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.913 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.914 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.915 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.916 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_list: /' 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_list.log 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.917 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.918 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.919 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.920 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82417/exe 2024-03-24T11:42:07.920 INFO:tasks.workunit.client.0.smithi114.stdout:test api_tier_pp on pid 82506 2024-03-24T11:42:07.924 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.924 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_tier_pp 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82506 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_tier_pp on pid 82506' 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82506 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.925 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_lock: /' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_pool 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_pool' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.926 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.927 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.928 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.929 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.930 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.931 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82439/exe 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.932 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_pool 2024-03-24T11:42:07.933 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.933 INFO:tasks.workunit.client.0.smithi114.stdout:test api_pool on pid 82543 2024-03-24T11:42:07.936 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_pool 2024-03-24T11:42:07.936 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82543 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_pool on pid 82543' 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82543 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.937 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.938 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.939 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.940 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_snapshots 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_snapshots' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82465/exe 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.941 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.942 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_snapshots 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.943 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.944 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.944 INFO:tasks.workunit.client.0.smithi114.stdout:test api_snapshots on pid 82602 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.949 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_snapshots 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82602 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_snapshots on pid 82602' 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82602 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_snapshots_pp 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_snapshots_pp' 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_misc: /' 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-24T11:42:07.951 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_snapshots_pp 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.952 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.953 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.954 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.955 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.956 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_snapshots_pp 2024-03-24T11:42:07.957 INFO:tasks.workunit.client.0.smithi114.stdout:test api_snapshots_pp on pid 82644 2024-03-24T11:42:07.964 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82644 2024-03-24T11:42:07.964 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_snapshots_pp on pid 82644' 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82644 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_stat 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.965 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.966 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.967 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_stat' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.968 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82484/exe 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.969 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.970 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.971 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.972 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.973 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82506/exe 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.974 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.975 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.976 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.977 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.978 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.979 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.980 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_stat 2024-03-24T11:42:07.981 INFO:tasks.workunit.client.0.smithi114.stdout:test api_stat on pid 82705 2024-03-24T11:42:07.984 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_stat 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82705 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_stat on pid 82705' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82705 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:07.985 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_stat_pp 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_stat_pp' 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82543/exe 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:07.986 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:07.987 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82602/exe 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.988 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:07.989 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.990 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.991 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.991 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.991 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_stat_pp 2024-03-24T11:42:07.991 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.991 INFO:tasks.workunit.client.0.smithi114.stdout:test api_stat_pp on pid 82740 2024-03-24T11:42:07.992 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_stat_pp 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_pool: /' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82740 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_stat_pp on pid 82740' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82740 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:07.993 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:07.994 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.995 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_watch_notify 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_watch_notify' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_watch_notify 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:07.996 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:07.997 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:07.998 INFO:tasks.workunit.client.0.smithi114.stdout:test api_watch_notify on pid 82759 2024-03-24T11:42:07.999 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_watch_notify 2024-03-24T11:42:07.999 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82759 2024-03-24T11:42:07.999 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_watch_notify on pid 82759' 2024-03-24T11:42:07.999 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82759 2024-03-24T11:42:07.999 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:07.999 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:07.999 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_watch_notify_pp 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_watch_notify_pp' 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_watch_notify_pp 2024-03-24T11:42:08.000 INFO:tasks.workunit.client.0.smithi114.stdout:test api_watch_notify_pp on pid 82767 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_watch_notify_pp 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82767 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_watch_notify_pp on pid 82767' 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82767 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_cmd 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_cmd' 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.004 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.005 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.006 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_cmd 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.007 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.008 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.009 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.010 INFO:tasks.workunit.client.0.smithi114.stdout:test api_cmd on pid 82785 2024-03-24T11:42:08.010 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_cmd 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82785 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_cmd on pid 82785' 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82785 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_cmd_pp 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_cmd_pp' 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_cmd_pp 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.011 INFO:tasks.workunit.client.0.smithi114.stdout:test api_cmd_pp on pid 82792 2024-03-24T11:42:08.015 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_cmd_pp 2024-03-24T11:42:08.015 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82792 2024-03-24T11:42:08.015 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_cmd_pp on pid 82792' 2024-03-24T11:42:08.015 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82792 2024-03-24T11:42:08.015 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.016 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.017 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.018 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.019 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.020 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.021 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.022 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.023 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_service 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_service' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.024 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.025 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_service 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.026 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.027 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.028 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.029 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.029 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_service 2024-03-24T11:42:08.029 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82830 2024-03-24T11:42:08.029 INFO:tasks.workunit.client.0.smithi114.stdout:test api_service on pid 82830 2024-03-24T11:42:08.034 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_service on pid 82830' 2024-03-24T11:42:08.034 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82830 2024-03-24T11:42:08.034 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.034 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.034 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_service_pp 2024-03-24T11:42:08.034 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82644/exe 2024-03-24T11:42:08.034 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_service_pp' 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_service_pp 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.035 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.036 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.037 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.038 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82705/exe 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.039 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.040 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.041 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.042 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.043 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.043 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.043 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.043 INFO:tasks.workunit.client.0.smithi114.stdout:test api_service_pp on pid 82865 2024-03-24T11:42:08.050 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.050 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_service_pp 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.051 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82865 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_service_pp on pid 82865' 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82865 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-24T11:42:08.052 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.053 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.054 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82759/exe 2024-03-24T11:42:08.055 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.056 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_stat: /' 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.057 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_c_write_operations 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_c_write_operations' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-24T11:42:08.058 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.059 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.060 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.061 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82740/exe 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_c_write_operations 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.062 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-24T11:42:08.063 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.064 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.065 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.066 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.066 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.066 INFO:tasks.workunit.client.0.smithi114.stdout:test api_c_write_operations on pid 82940 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_c_write_operations 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82940 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_c_write_operations on pid 82940' 2024-03-24T11:42:08.071 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82940 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.072 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82767/exe 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_c_read_operations 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_c_read_operations' 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.073 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.074 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.075 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.076 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.077 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.078 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82792/exe 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.079 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_c_read_operations 2024-03-24T11:42:08.080 INFO:tasks.workunit.client.0.smithi114.stdout:test api_c_read_operations on pid 82988 2024-03-24T11:42:08.081 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.082 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_c_read_operations 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82988 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_c_read_operations on pid 82988' 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=82988 2024-03-24T11:42:08.083 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s api_cls_remote_reads 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.084 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' api_cls_remote_reads' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.085 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo api_cls_remote_reads 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.086 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=api_cls_remote_reads 2024-03-24T11:42:08.087 INFO:tasks.workunit.client.0.smithi114.stdout:test api_cls_remote_reads on pid 83002 2024-03-24T11:42:08.090 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83002 2024-03-24T11:42:08.090 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test api_cls_remote_reads on pid 83002' 2024-03-24T11:42:08.090 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83002 2024-03-24T11:42:08.090 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.090 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.090 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s list_parallel 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' list_parallel' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.091 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.092 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.093 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82830/exe 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.094 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.095 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.096 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.097 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.098 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo list_parallel 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.099 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.100 INFO:tasks.workunit.client.0.smithi114.stdout:test list_parallel on pid 83035 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=list_parallel 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83035 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test list_parallel on pid 83035' 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83035 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.104 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s open_pools_parallel 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' open_pools_parallel' 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82785/exe 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.105 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_service: /' 2024-03-24T11:42:08.106 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo open_pools_parallel 2024-03-24T11:42:08.107 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_service.log 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=open_pools_parallel 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83052 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test open_pools_parallel on pid 83052' 2024-03-24T11:42:08.108 INFO:tasks.workunit.client.0.smithi114.stdout:test open_pools_parallel on pid 83052 2024-03-24T11:42:08.113 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83052 2024-03-24T11:42:08.113 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-24T11:42:08.113 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.113 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.113 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s delete_pools_parallel 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.114 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' delete_pools_parallel' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.115 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo delete_pools_parallel 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_cmd: /' 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82865/exe 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.116 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.117 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.118 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.119 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.120 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.120 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.120 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.120 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.120 INFO:tasks.workunit.client.0.smithi114.stdout:test delete_pools_parallel on pid 83093 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=delete_pools_parallel 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83093 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test delete_pools_parallel on pid 83093' 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83093 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-24T11:42:08.125 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.126 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.127 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.128 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.129 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s cls 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' cls' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.130 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo cls 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.131 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.132 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.133 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.134 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83002/exe 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82940/exe 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.135 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.136 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.137 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-24T11:42:08.138 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=cls 2024-03-24T11:42:08.139 INFO:tasks.workunit.client.0.smithi114.stdout:test cls on pid 83163 2024-03-24T11:42:08.143 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83163 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test cls on pid 83163' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83163 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s cmd 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' cmd' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.144 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.145 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.146 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/82988/exe 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.147 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.148 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo cmd 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.149 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.150 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83035/exe 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.151 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.152 INFO:tasks.workunit.client.0.smithi114.stdout:test cmd on pid 83199 2024-03-24T11:42:08.158 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.158 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.158 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=cmd 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83199 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test cmd on pid 83199' 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83199 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-24T11:42:08.159 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ list_parallel: /' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.160 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s handler_error 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' handler_error' 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.161 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.162 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.163 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.164 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo handler_error 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.165 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.166 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.167 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.168 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83052/exe 2024-03-24T11:42:08.169 INFO:tasks.workunit.client.0.smithi114.stdout:test handler_error on pid 83237 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=handler_error 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83237 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test handler_error on pid 83237' 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83237 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.172 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s io 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' io' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.173 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo io 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.174 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.175 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.176 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.177 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.178 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.179 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.180 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.181 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.182 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.182 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.182 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.182 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.182 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.182 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.182 INFO:tasks.workunit.client.0.smithi114.stdout:test io on pid 83291 2024-03-24T11:42:08.183 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=io 2024-03-24T11:42:08.183 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83291 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test io on pid 83291' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83291 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s ec_io 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' ec_io' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.184 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo ec_io 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.185 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.186 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.187 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.187 INFO:tasks.workunit.client.0.smithi114.stdout:test ec_io on pid 83331 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=ec_io 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83331 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test ec_io on pid 83331' 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83331 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.191 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.192 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s list 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' list' 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo list 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.193 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.194 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83163/exe 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.195 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.196 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.197 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.198 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.199 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_cls 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83093/exe 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.200 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.201 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ cls: /' 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.202 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_cls.log 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-24T11:42:08.203 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83199/exe 2024-03-24T11:42:08.204 INFO:tasks.workunit.client.0.smithi114.stdout:test list on pid 83418 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=list 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83418 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test list on pid 83418' 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83418 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.208 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-24T11:42:08.209 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.210 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_cmd 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s ec_list 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' ec_list' 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ cmd: /' 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83237/exe 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.211 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_handler_error 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.212 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ handler_error: /' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.213 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo ec_list 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.214 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.215 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.216 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.217 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.217 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.217 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.217 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.217 INFO:tasks.workunit.client.0.smithi114.stdout:test ec_list on pid 83496 2024-03-24T11:42:08.219 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=ec_list 2024-03-24T11:42:08.219 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83496 2024-03-24T11:42:08.219 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test ec_list on pid 83496' 2024-03-24T11:42:08.219 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83496 2024-03-24T11:42:08.219 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.219 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.219 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83291/exe 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.220 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_io.log 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.221 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ io: /' 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_io 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s misc 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' misc' 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-24T11:42:08.222 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo misc 2024-03-24T11:42:08.223 INFO:tasks.workunit.client.0.smithi114.stdout:test misc on pid 83537 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.225 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=misc 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83537 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test misc on pid 83537' 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83537 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s pool 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' pool' 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo pool 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.226 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.227 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.229 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.230 INFO:tasks.workunit.client.0.smithi114.stdout:test pool on pid 83565 2024-03-24T11:42:08.233 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=pool 2024-03-24T11:42:08.233 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83565 2024-03-24T11:42:08.233 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test pool on pid 83565' 2024-03-24T11:42:08.233 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83565 2024-03-24T11:42:08.233 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.233 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s read_operations 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' read_operations' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.234 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.235 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.236 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.237 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo read_operations 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.238 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83331/exe 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.239 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.240 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.241 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.242 INFO:tasks.workunit.client.0.smithi114.stdout:test read_operations on pid 83607 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=read_operations 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83607 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test read_operations on pid 83607' 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83607 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.246 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s snapshots 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' snapshots' 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.247 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo snapshots 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.248 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_ec_io 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.249 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.250 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83418/exe 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.251 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.252 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ ec_io: /' 2024-03-24T11:42:08.253 INFO:tasks.workunit.client.0.smithi114.stdout:test snapshots on pid 83644 2024-03-24T11:42:08.254 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=snapshots 2024-03-24T11:42:08.254 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83644 2024-03-24T11:42:08.254 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test snapshots on pid 83644' 2024-03-24T11:42:08.254 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83644 2024-03-24T11:42:08.254 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s watch_notify 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' watch_notify' 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.255 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_list 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo watch_notify 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.256 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.257 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.258 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.259 INFO:tasks.workunit.client.0.smithi114.stdout:test watch_notify on pid 83677 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=watch_notify 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83677 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test watch_notify on pid 83677' 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83677 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.262 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.264 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:++ printf %25s write_operations 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+ r=' write_operations' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.265 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.266 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ echo write_operations 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.267 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++ awk '{print $1}' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.268 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_list.log 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.269 INFO:tasks.workunit.client.0.smithi114.stdout:test write_operations on pid 83701 2024-03-24T11:42:08.283 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.283 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.283 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ ff=write_operations 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83701 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'test write_operations on pid 83701' 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ pids[$f]=83701 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ ret=0 2024-03-24T11:42:08.284 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 1 -eq 1 ']' 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82506 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82506 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83537/exe 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.285 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.286 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.287 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.288 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -f /etc/bashrc ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:+ . /etc/bashrc 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83496/exe 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.289 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -z '' ']' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ BASHRCSOURCED=Y 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ shopt -q login_shell 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ list: /' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83565/exe 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.290 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.291 INFO:tasks.workunit.client.0.smithi114.stderr:+++ umask 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' 0022 -eq 0 ']' 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ SHELL=/bin/bash 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.292 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.293 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_misc 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.294 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_misc.log 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_pool 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ misc: /' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ ec_list: /' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.295 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_ec_list 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.296 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_pool.log 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ pool: /' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.297 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.298 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.299 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.300 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83607/exe 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.301 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.302 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.303 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'grep=grep --color=auto' 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83677/exe 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorls.sh 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' '!' -t 0 ']' 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:+++ return 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.304 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export S_COLORS=auto 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:+++ S_COLORS=auto 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ read_operations: /' 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.305 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_read_operations 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.306 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.307 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ watch_notify: /' 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_watch_notify 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.308 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83644/exe 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-24T11:42:08.309 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.310 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.313 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.313 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.313 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_snapshots 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ snapshots: /' 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-24T11:42:08.314 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/gawk.sh 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/lang.sh 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/locale 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.315 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:++++ export LANG=en_US.UTF-8 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:++++ LANG=en_US.UTF-8 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ LANG=en_US.UTF-8 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ unset LANG_backup config 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n '' ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' dumb = linux ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.316 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/less.sh 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:++ for i in /etc/profile.d/*.sh 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:++ '[' '' ']' 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:++ . /etc/profile.d/which2.sh 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+++++ readlink /proc/83701/exe 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:++++ basename /usr/bin/bash 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_declare='declare -f' 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+++ which_opt=-f 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export which_declare 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+++ export -f which 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset i 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:++ unset -f pathmunge 2024-03-24T11:42:08.317 INFO:tasks.workunit.client.0.smithi114.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-24T11:42:08.318 INFO:tasks.workunit.client.0.smithi114.stderr:+ export PATH 2024-03-24T11:42:08.318 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-24T11:42:08.318 INFO:tasks.workunit.client.0.smithi114.stderr:+ unset rc 2024-03-24T11:42:08.318 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph_test_neorados_write_operations 2024-03-24T11:42:08.318 INFO:tasks.workunit.client.0.smithi114.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-24T11:42:08.318 INFO:tasks.workunit.client.0.smithi114.stderr:+ sed 's/^/ write_operations: /' 2024-03-24T11:42:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3843810818' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-24T11:42:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3843810818' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-24T11:42:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3843810818' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-24T11:42:11.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:42:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:42:13.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2662146454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2723749760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:42:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:12 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2662146454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2723749760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2662146454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2723749760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:12 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:42:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:42:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1689599136' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1877172033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3653771578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: pgmap v70: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:42:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: pgmap v71: 356 pgs: 149 creating+peering, 154 active+clean, 53 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1397119133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi114-82444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.34745 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.34661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[42466]: from='client.24664 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1689599136' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1877172033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3653771578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: pgmap v70: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:42:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: pgmap v71: 356 pgs: 149 creating+peering, 154 active+clean, 53 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1397119133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi114-82444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.34745 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.34661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:16.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:16 smithi114 ceph-mon[30312]: from='client.24664 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1689599136' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1877172033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3653771578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: pgmap v70: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:42:17.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: pgmap v71: 356 pgs: 149 creating+peering, 154 active+clean, 53 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1397119133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi114-82444-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.34745 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.34661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:17.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:16 smithi119 ceph-mon[33714]: from='client.24664 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:17.304 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [----------] Global test environment set-up. 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5076 ms) 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ OK ] LibRadosService.RegisterLate (28 ms) 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: cluster: 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: id: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: health: HEALTH_OK 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: services: 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-24T11:42:17.305 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: data: 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: usage: 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pgs: 132 active+clean 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: io: 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: cluster: 2024-03-24T11:42:17.306 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: id: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: health: HEALTH_OK 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: services: 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: data: 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: usage: 230 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pgs: 132 active+clean 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.307 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: io: 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: cluster: 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: id: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: health: HEALTH_WARN 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 7 pool(s) do not have an application enabled 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: services: 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.308 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: data: 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pools: 13 pools, 356 pgs 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pgs: 14.888% pgs unknown 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 41.854% pgs not active 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 154 active+clean 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 149 creating+peering 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 53 unknown 2024-03-24T11:42:17.309 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [----------] Global test environment set-up. 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (39 ms) 2024-03-24T11:42:17.934 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (8 ms) 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4 ms) 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (2 ms) 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-24T11:42:17.935 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (5 ms) 2024-03-24T11:42:17.936 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [----------] 12 tests from AsioRados (66 ms total) 2024-03-24T11:42:17.936 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: 2024-03-24T11:42:17.936 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [----------] Global test environment tear-down 2024-03-24T11:42:17.936 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (10057 ms total) 2024-03-24T11:42:17.936 INFO:tasks.workunit.client.0.smithi114.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-24T11:42:18.261 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] Global test environment set-up. 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosList.ListObjects (374 ms) 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (42 ms) 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset foo1,foo2,foo3 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo1 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo2 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo3 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset foo1,foo4,foo5 2024-03-24T11:42:18.262 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo4 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo5 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo1 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset foo6,foo7 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo7 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo6 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns1:foo4 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns1:foo5 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns2:foo7 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns2:foo6 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns1:foo1 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: :foo1 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: :foo2 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: :foo3 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (79 ms) 2024-03-24T11:42:18.263 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 1 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 10 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 13 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 7 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 14 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 0 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 15 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 11 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 5 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 8 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 6 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 3 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 4 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 12 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 9 0 2024-03-24T11:42:18.264 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 2 0 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (64 ms) 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: x cursor=MIN 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=1 cursor=14:02547ec2:::1:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=10 cursor=14:52ea6a34:::10:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=13 cursor=14:566253c9:::13:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=7 cursor=14:5c6b0b28:::7:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=14 cursor=14:62a1935d:::14:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=0 cursor=14:6cac518f:::0:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=15 cursor=14:863748b0:::15:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=11 cursor=14:89d3ae78:::11:head 2024-03-24T11:42:18.265 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=5 cursor=14:b29083e3:::5:head 2024-03-24T11:42:18.266 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=8 cursor=14:bd63b0f1:::8:head 2024-03-24T11:42:18.266 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=6 cursor=14:c4fdafeb:::6:head 2024-03-24T11:42:18.266 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=3 cursor=14:cfc208b3:::3:head 2024-03-24T11:42:18.266 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=4 cursor=14:d83876eb:::4:head 2024-03-24T11:42:18.266 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=12 cursor=14:de5d7c5f:::12:head 2024-03-24T11:42:18.266 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=9 cursor=14:e960b815:::9:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > oid=2 cursor=14:f905c69b:::2:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=1 cursor=14:02547ec2:::1:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:02547ec2:::1:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:02547ec2:::1:head -> 1 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=10 cursor=14:52ea6a34:::10:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:52ea6a34:::10:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:52ea6a34:::10:head -> 10 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=13 cursor=14:566253c9:::13:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:566253c9:::13:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:566253c9:::13:head -> 13 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=7 cursor=14:5c6b0b28:::7:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:5c6b0b28:::7:head 2024-03-24T11:42:18.267 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:5c6b0b28:::7:head -> 7 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=14 cursor=14:62a1935d:::14:head 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:62a1935d:::14:head 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:62a1935d:::14:head -> 14 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=0 cursor=14:6cac518f:::0:head 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:6cac518f:::0:head 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:6cac518f:::0:head -> 0 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=15 cursor=14:863748b0:::15:head 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:863748b0:::15:head 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:863748b0:::15:head -> 15 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=11 cursor=14:89d3ae78:::11:head 2024-03-24T11:42:18.316 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:89d3ae78:::11:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:89d3ae78:::11:head -> 11 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=5 cursor=14:b29083e3:::5:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:b29083e3:::5:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:b29083e3:::5:head -> 5 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=8 cursor=14:bd63b0f1:::8:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:bd63b0f1:::8:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:bd63b0f1:::8:head -> 8 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=6 cursor=14:c4fdafeb:::6:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:c4fdafeb:::6:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:c4fdafeb:::6:head -> 6 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=3 cursor=14:cfc208b3:::3:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:cfc208b3:::3:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:cfc208b3:::3:head -> 3 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=4 cursor=14:d83876eb:::4:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:d83876eb:::4:head 2024-03-24T11:42:18.317 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:d83876eb:::4:head -> 4 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=12 cursor=14:de5d7c5f:::12:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:de5d7c5f:::12:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:de5d7c5f:::12:head -> 12 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=9 cursor=14:e960b815:::9:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:e960b815:::9:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:e960b815:::9:head -> 9 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : oid=2 cursor=14:f905c69b:::2:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:f905c69b:::2:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:f905c69b:::2:head -> 2 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:f905c69b:::2:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:f905c69b:::2:head expected=14:f905c69b:::2:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:f905c69b:::2:head -> 2 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=2 expected=2 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:cfc208b3:::3:head 2024-03-24T11:42:18.318 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:cfc208b3:::3:head expected=14:cfc208b3:::3:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:cfc208b3:::3:head -> 3 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=3 expected=3 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:c4fdafeb:::6:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:c4fdafeb:::6:head expected=14:c4fdafeb:::6:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:c4fdafeb:::6:head -> 6 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=6 expected=6 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:863748b0:::15:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:863748b0:::15:head expected=14:863748b0:::15:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:863748b0:::15:head -> 15 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=15 expected=15 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:e960b815:::9:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:e960b815:::9:head expected=14:e960b815:::9:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:e960b815:::9:head -> 9 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=9 expected=9 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:de5d7c5f:::12:head 2024-03-24T11:42:18.319 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:de5d7c5f:::12:head expected=14:de5d7c5f:::12:head 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:de5d7c5f:::12:head -> 12 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=12 expected=12 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:d83876eb:::4:head 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:d83876eb:::4:head expected=14:d83876eb:::4:head 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:d83876eb:::4:head -> 4 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=4 expected=4 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:bd63b0f1:::8:head 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:bd63b0f1:::8:head expected=14:bd63b0f1:::8:head 2024-03-24T11:42:18.320 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:bd63b0f1:::8:head -> 8 2024-03-24T11:42:18.481 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=8 list_parallel: process_1_[83241]: starting. 2024-03-24T11:42:18.481 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_1_[83241]: creating pool ceph_test_rados_list_parallel.smithi114-83201 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_1_[83241]: created object 0... 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_1_[83241]: created object 25... 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_1_[83241]: created object 49... 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_1_[83241]: finishing. 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_1_[83241]: shutting down. 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_2_[83244]: starting. 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_2_[83244]: listing objects. 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_2_[83244]: listed object 0... 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_2_[83244]: listed object 25... 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_2_[83244]: saw 50 objects 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_2_[83244]: shutting down. 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_3_[84409]: starting. 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_3_[84409]: creating pool ceph_test_rados_list_parallel.smithi114-83201 2024-03-24T11:42:18.482 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_3_[84409]: created object 0... 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_3_[84409]: created object 25... 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_3_[84409]: created object 49... 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_3_[84409]: finishing. 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_3_[84409]: shutting down. 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_4_[84410]: starting. 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_4_[84410]: listing objects. 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_4_[84410]: listed object 0... 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_4_[84410]: listed object 25... 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_4_[84410]: saw 43 objects 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_4_[84410]: shutting down. 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_5_[84411]: starting. 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_5_[84411]: removed 25 objects... 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_5_[84411]: removed half of the objects 2024-03-24T11:42:18.483 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_5_[84411]: removed 50 objects... 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_5_[84411]: removed 50 objects 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_5_[84411]: shutting down. 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_6_[84457]: starting. 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_6_[84457]: creating pool ceph_test_rados_list_parallel.smithi114-83201 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_6_[84457]: created object 0... 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_6_[84457]: created object 25... 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_6_[84457]: created object 49... 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_6_[84457]: finishing. 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_6_[84457]: shutting down. 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_7_[84458]: starting. 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_7_[84458]: listing objects. 2024-03-24T11:42:18.484 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_7_[84458]: listed object 0... 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_7_[84458]: listed object 25... 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_7_[84458]: listed object 50... 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_7_[84458]: saw 56 objects 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_7_[84458]: shutting down. 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_8_[84459]: starting. 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_8_[84459]: added 25 objects... 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_8_[84459]: added half of the objects 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_8_[84459]: added 50 objects... 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_8_[84459]: added 50 objects 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_8_[84459]: shutting down. 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.485 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2459035233' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2203694547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4195584434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4029256223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2104391213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1939566289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2957324817' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: pgmap v73: 772 pgs: 149 creating+peering, 154 active+clean, 469 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2107349620' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: pgmap v74: 772 pgs: 449 creating+peering, 154 active+clean, 169 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:18.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.34775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1397119133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi114-82444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.34745 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.34661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24664 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/804570541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1216479313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3774563794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/889282923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/431188569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/253530869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24727 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24742 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.24745 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4111182494' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:18.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2459035233' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2203694547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4195584434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4029256223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2104391213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1939566289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2957324817' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: pgmap v73: 772 pgs: 149 creating+peering, 154 active+clean, 469 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2107349620' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: pgmap v74: 772 pgs: 449 creating+peering, 154 active+clean, 169 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:42:18.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.34775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1397119133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi114-82444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.34745 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.34661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24664 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/804570541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1216479313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3774563794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/889282923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/431188569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/253530869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24727 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24742 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.24745 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4111182494' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi114-83598-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2459035233' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2203694547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4195584434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4029256223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2104391213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1939566289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2957324817' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: pgmap v73: 772 pgs: 149 creating+peering, 154 active+clean, 469 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2107349620' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: pgmap v74: 772 pgs: 449 creating+peering, 154 active+clean, 169 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.34775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1397119133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi114-82444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.34745 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi114-82935-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.34661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi114-82433-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi114-83716-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24664 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi114-82405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:18.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/804570541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1216479313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3774563794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/889282923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/431188569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/253530869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24727 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24742 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.24745 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:18.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4111182494' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:18.811 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.811 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_9_[84548]: starting. 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_9_[84548]: creating pool ceph_test_rados_list_parallel.smithi114-83201 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_9_[84548]: created object 0... 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_9_[84548]: created object 25... 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_9_[84548]: created object 49... 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_9_[84548]: finishing. 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_9_[84548]: shutting down. 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: starting. 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: listing objects. 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: listed object 0... 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: listed object 25... 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: listed object 50... 2024-03-24T11:42:18.812 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: listed object 75... 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: saw 98 objects 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_10_[84549]: shutting down. 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_13_[84552]: starting. 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_13_[84552]: removed 25 objects... 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_13_[84552]: removed half of the objects 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_13_[84552]: removed 50 objects... 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_13_[84552]: removed 50 objects 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_13_[84552]: shutting down. 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_11_[84550]: starting. 2024-03-24T11:42:18.813 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_11_[84550]: added 25 objects... 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_11_[84550]: added half of the objects 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_11_[84550]: added 50 objects... 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_11_[84550]: added 50 objects 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_11_[84550]: shutting down. 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_12_[84551]: starting. 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_12_[84551]: added 25 objects... 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_12_[84551]: added half of the objects 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_12_[84551]: added 50 objects... 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_12_[84551]: added 50 objects 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_12_[84551]: shutting down. 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.814 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_14_[84628]: starting. 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_14_[84628]: creating pool ceph_test_rados_list_parallel.smithi114-83201 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_14_[84628]: created object 0... 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_14_[84628]: created object 25... 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_14_[84628]: created object 49... 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_14_[84628]: finishing. 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_14_[84628]: shutting down. 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: starting. 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: listing objects. 2024-03-24T11:42:18.815 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: listed object 0... 2024-03-24T11:42:18.902 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: listed object 2 handler_error: Running main() from gmock_main.cc 2024-03-24T11:42:18.902 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-24T11:42:18.903 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [----------] Global test environment set-up. 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (10713 ms) 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [----------] 1 test from neocls_handler_error (10713 ms total) 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [----------] Global test environment tear-down 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [==========] 1 test from 1 test suite ran. (10713 ms total) 2024-03-24T11:42:18.904 INFO:tasks.workunit.client.0.smithi114.stdout: handler_error: [ PASSED ] 1 test. 2024-03-24T11:42:18.908 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-24T11:42:18.908 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-24T11:42:18.908 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-24T11:42:18.908 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (299 ms) 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (10 ms) 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (25 ms) 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (36 ms) 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-24T11:42:18.909 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (7 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (5 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (7 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (9 ms) 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-24T11:42:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-03-24T11:42:18.911 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-24T11:42:18.911 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5 ms) 2024-03-24T11:42:18.911 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (454 ms total) 2024-03-24T11:42:18.911 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: 2024-03-24T11:42:18.911 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-24T11:42:18.911 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (10820 ms total) 2024-03-24T11:42:18.911 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-24T11:42:19.750 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (11659 ms) 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (11659 ms total) 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (11659 ms total) 2024-03-24T11:42:19.751 INFO:tasks.workunit.client.0.smithi114.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-24T11:42:19.767 INFO:tasks.workunit.client.0.smithi114.stdout:5... 2024-03-24T11:42:19.767 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: listed object 50... 2024-03-24T11:42:19.767 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: listed object 75... 2024-03-24T11:42:19.767 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: listed object 100... 2024-03-24T11:42:19.767 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: listed object 125... 2024-03-24T11:42:19.767 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: saw 150 objects 2024-03-24T11:42:19.767 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_15_[84629]: shutting down. 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_16_[84630]: starting. 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_16_[84630]: added 25 objects... 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_16_[84630]: added half of the objects 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_16_[84630]: added 50 objects... 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_16_[84630]: added 50 objects 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: process_16_[84630]: shutting down. 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******************************* 2024-03-24T11:42:19.768 INFO:tasks.workunit.client.0.smithi114.stdout: list_parallel: ******* SUCCESS ********** 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.34775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24727 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24745 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: pgmap v77: 1156 pgs: 397 creating+peering, 142 active+clean, 617 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/364348968' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.34775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24727 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24745 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: pgmap v77: 1156 pgs: 397 creating+peering, 142 active+clean, 617 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/364348968' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:19 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:20.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi114-82459-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.34775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi114-83187-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi114-82508-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi114-82498-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi114-82839-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi114-82548-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi114-82736-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24727 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi114-82677-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi114-82855-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82993-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83142-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83045-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24745 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi114-82896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: pgmap v77: 1156 pgs: 397 creating+peering, 142 active+clean, 617 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi114-83598-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi114-83716-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/364348968' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:20.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:19 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:21.253 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-24T11:42:21.253 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-24T11:42:21.253 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-24T11:42:21.253 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-24T11:42:21.253 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-24T11:42:21.253 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (59 ms) 2024-03-24T11:42:21.253 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (77 ms) 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (13099 ms) 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (13235 ms total) 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (13235 ms total) 2024-03-24T11:42:21.254 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/225526481' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/225526481' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:21 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:21.531 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:21.531 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/225526481' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-24T11:42:21.531 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:21.531 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:21 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:21.531 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:21 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:21.531 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:21 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:21.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:42:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (68 ms) 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (55 ms) 2024-03-24T11:42:22.237 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (13089 ms) 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.178495+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.688137+0000 mon.a [INF] from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.693682+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.730231+0000 mon.a [INF] from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi114-82958-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.730310+0000 mon.a [INF] from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.737648+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.738252+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.738994+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.238 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.757702+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:19.813548+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.269763+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.270398+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.270893+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.271294+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.271708+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.272121+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.272523+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.272884+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.273242+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:20.273599+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:22.239 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.221970+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.226969+0000 mon.a [INF] from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.237786+0000 mon.a [INF] from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.247994+0000 mon.a [INF] from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.248526+0000 mon.a [INF] from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.252205+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/4252475157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.313795+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.327257+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.327556+0000 client.admin [INF] onexx 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.524240+0000 mon.a [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.240 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.524794+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:22.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:22.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:22.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.8 deep-scrub starts 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.8 deep-scrub ok 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.2 deep-scrub starts 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.2 deep-scrub ok 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.7 deep-scrub starts 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.7 deep-scrub ok 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: pgmap v80: 852 pgs: 194 creating+peering, 142 active+clean, 516 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:22.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.6 deep-scrub starts 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.6 deep-scrub ok 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.1 deep-scrub starts 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.1 deep-scrub ok 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.3 deep-scrub starts 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.3 deep-scrub ok 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4252475157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: onexx 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.4 deep-scrub starts 2024-03-24T11:42:22.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.4 deep-scrub ok 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.8 deep-scrub starts 2024-03-24T11:42:22.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.8 deep-scrub ok 2024-03-24T11:42:22.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.2 deep-scrub starts 2024-03-24T11:42:22.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.2 deep-scrub ok 2024-03-24T11:42:22.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.7 deep-scrub starts 2024-03-24T11:42:22.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.7 deep-scrub ok 2024-03-24T11:42:22.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: pgmap v80: 852 pgs: 194 creating+peering, 142 active+clean, 516 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:22.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:22.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.6 deep-scrub starts 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.6 deep-scrub ok 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.1 deep-scrub starts 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.1 deep-scrub ok 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.3 deep-scrub starts 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.3 deep-scrub ok 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4252475157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: onexx 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.4 deep-scrub starts 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.4 deep-scrub ok 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.25162 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:22.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[42466]: 22.5 deep-scrub starts 2024-03-24T11:42:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.25162 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:22 smithi114 ceph-mon[30312]: 22.5 deep-scrub starts 2024-03-24T11:42:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi114-82405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.8 deep-scrub starts 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.8 deep-scrub ok 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.2 deep-scrub starts 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.2 deep-scrub ok 2024-03-24T11:42:22.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.7 deep-scrub starts 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.7 deep-scrub ok 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: pgmap v80: 852 pgs: 194 creating+peering, 142 active+clean, 516 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.6 deep-scrub starts 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.6 deep-scrub ok 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.1 deep-scrub starts 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.1 deep-scrub ok 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.3 deep-scrub starts 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.3 deep-scrub ok 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:42:22.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4252475157' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: onexx 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.4 deep-scrub starts 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.4 deep-scrub ok 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.25162 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:22.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:22 smithi119 ceph-mon[33714]: 22.5 deep-scrub starts 2024-03-24T11:42:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: 22.9 deep-scrub starts 2024-03-24T11:42:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: 22.9 deep-scrub ok 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: twoxx 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.25162 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:23.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:42:23.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:23.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3584490768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:23.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:23.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: 22.9 deep-scrub starts 2024-03-24T11:42:23.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: 22.9 deep-scrub ok 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: twoxx 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.25162 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3584490768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:23.744 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: pgmap v83: 812 pgs: 200 creating+peering, 516 active+clean, 96 unknown; 145 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:23.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: pgmap v83: 812 pgs: 200 creating+peering, 516 active+clean, 96 unknown; 145 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:23.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:23.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:23.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:23.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:23 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: 22.9 deep-scrub starts 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: 22.9 deep-scrub ok 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: twoxx 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi114-82444-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.25162 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3584490768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:23.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/734421220' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: pgmap v83: 812 pgs: 200 creating+peering, 516 active+clean, 96 unknown; 145 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:23.807 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.25237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:23.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:23 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:24 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-24T11:42:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:24 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-24T11:42:24.990 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:42:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:42:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:25.116 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:25.116 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:24 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:21.762083+0000 mon.a [WRN] Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.002109+0000 mon.a [INF] from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.002306+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.002533+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.002665+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.003000+0000 mon.a [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.003305+0000 mon.a [INF] from='client.25162 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:25.859 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.003322+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.003418+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.003521+0000 mon.a [INF] from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.003860+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.003899+0000 mon.a [INF] from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi114-82855-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.004265+0000 mon.a [INF] from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:22.005424+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.102513+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.345737+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.352924+0000 mon.a [INF] from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.352989+0000 mon.a [INF] from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.353064+0000 mon.a [INF] from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.361175+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.390108+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:23.883484+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:25.860 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.125733+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.862126+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.862205+0000 mon.a [INF] from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.862260+0000 mon.a [INF] from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.862330+0000 mon.a [INF] from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.874444+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/556207526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.881601+0000 mon.a [INF] from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.882102+0000 mon.a [INF] from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.861 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:24.883550+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:25.881 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.140176+0000 mon.a [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool" cls: Running main() from gmock_main.cc 2024-03-24T11:42:25.881 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-24T11:42:25.881 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [----------] Global test environment set-up. 2024-03-24T11:42:25.881 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-24T11:42:25.881 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-24T11:42:25.881 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [ OK ] NeoRadosCls.DNE (10740 ms) 2024-03-24T11:42:25.882 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-24T11:42:25.882 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (6983 ms) 2024-03-24T11:42:25.882 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [----------] 2 tests from NeoRadosCls (17723 ms total) 2024-03-24T11:42:25.882 INFO:tasks.workunit.client.0.smithi114.stdout: cls: 2024-03-24T11:42:25.882 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [----------] Global test environment tear-down 2024-03-24T11:42:25.882 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [==========] 2 tests from 1 test suite ran. (17723 ms total) 2024-03-24T11:42:25.882 INFO:tasks.workunit.client.0.smithi114.stdout: cls: [ PASSED ] 2 tests. 2024-03-24T11:42:25.889 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_1_[83273]: starting. 2024-03-24T11:42:25.889 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_1_[83273]: creating pool ceph_test_rados_open_pools_parallel.smithi114-83240 2024-03-24T11:42:25.889 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_1_[83273]: created object 0... 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_1_[83273]: created object 25... 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_1_[83273]: created object 49... 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_1_[83273]: finishing. 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_1_[83273]: shutting down. 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_2_[83276]: starting. 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_2_[83276]: rados_pool_create. 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_2_[83276]: rados_ioctx_create. 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_2_[83276]: shutting down. 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: ******************************* 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_3_[84710]: starting. 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_3_[84710]: creating pool ceph_test_rados_open_pools_parallel.smithi114-83240 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_3_[84710]: created object 0... 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_3_[84710]: created object 25... 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_3_[84710]: created object 49... 2024-03-24T11:42:25.890 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_3_[84710]: finishing. 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_3_[84710]: shutting down. 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: ******************************* 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_4_[84712]: starting. 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_4_[84712]: rados_pool_create. 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_4_[84712]: rados_ioctx_create. 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: process_4_[84712]: shutting down. 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: ******************************* 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: ******************************* 2024-03-24T11:42:25.891 INFO:tasks.workunit.client.0.smithi114.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_1_[83478]: starting. 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_1_[83478]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-83408 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_1_[83478]: created object 0... 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_1_[83478]: created object 25... 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_1_[83478]: created object 49... 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_1_[83478]: finishing. 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_1_[83478]: shutting down. 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_2_[83480]: starting. 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_2_[83480]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-83408 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_2_[83480]: shutting down. 2024-03-24T11:42:25.909 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: ******************************* 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_3_[84709]: starting. 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_3_[84709]: creating pool ceph_test_rados_delete_pools_parallel.smithi114-83408 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_3_[84709]: created object 0... 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_3_[84709]: created object 25... 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_3_[84709]: created object 49... 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_3_[84709]: finishing. 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_3_[84709]: shutting down. 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: ******************************* 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_5_[84713]: starting. 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_5_[84713]: listing objects. 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_5_[84713]: listed object 0... 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_5_[84713]: listed object 25... 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_5_[84713]: saw 50 objects 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_5_[84713]: shutting down. 2024-03-24T11:42:25.910 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: ******************************* 2024-03-24T11:42:25.911 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_4_[84711]: starting. 2024-03-24T11:42:25.911 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_4_[84711]: deleting pool ceph_test_rados_delete_pools_parallel.smithi114-83408 2024-03-24T11:42:25.911 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: process_4_[84711]: shutting down. 2024-03-24T11:42:25.911 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: ******************************* 2024-03-24T11:42:25.911 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: ******************************* 2024-03-24T11:42:25.911 INFO:tasks.workunit.client.0.smithi114.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: seed 82459 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-24T11:42:26.110 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (307 ms) 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (11 ms) 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (7 ms) 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (8 ms) 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (10 ms) 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-24T11:42:26.111 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (10 ms) 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (6 ms) 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (4 ms) 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-03-24T11:42:26.112 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (4 ms) 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (12 ms) 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (6 ms) 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-03-24T11:42:26.113 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (5 ms) 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (422 ms total) 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1199 ms) 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (23 ms) 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-03-24T11:42:26.114 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-24T11:42:26.115 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5 ms) 2024-03-24T11:42:26.115 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-24T11:42:26.115 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-03-24T11:42:26.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: 22.5 deep-scrub ok 2024-03-24T11:42:26.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: 22.0 deep-scrub starts 2024-03-24T11:42:26.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: 22.0 deep-scrub ok 2024-03-24T11:42:26.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:26.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:26.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: pgmap v85: 876 pgs: 168 creating+peering, 484 active+clean, 224 unknown; 145 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 32 MiB/s wr, 467 op/s 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:26.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/556207526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: threexx 2024-03-24T11:42:26.153 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:26.153 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.153 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.153 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.153 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.153 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:26.153 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:25 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: 22.5 deep-scrub ok 2024-03-24T11:42:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: 22.0 deep-scrub starts 2024-03-24T11:42:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: 22.0 deep-scrub ok 2024-03-24T11:42:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: pgmap v85: 876 pgs: 168 creating+peering, 484 active+clean, 224 unknown; 145 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 32 MiB/s wr, 467 op/s 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/556207526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: 22.5 deep-scrub ok 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: 22.0 deep-scrub starts 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: 22.0 deep-scrub ok 2024-03-24T11:42:26.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi114-83598-1"}]': finished 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1909483764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: pgmap v85: 876 pgs: 168 creating+peering, 484 active+clean, 224 unknown; 145 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 32 MiB/s wr, 467 op/s 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi114-82935-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:26.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi114-82855-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi114-82459-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi114-82444-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/556207526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: threexx 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:26.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: threexx 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:26.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:25 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:27.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-24T11:42:27.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-24T11:42:27.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: fourxx 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:27.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2966322426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:27.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: pgmap v88: 804 pgs: 64 creating+peering, 740 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 28 KiB/s wr, 27 op/s 2024-03-24T11:42:27.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-24T11:42:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-24T11:42:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: fourxx 2024-03-24T11:42:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-24T11:42:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2966322426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: fourxx 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi114-83598-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:42:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2966322426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: pgmap v88: 804 pgs: 64 creating+peering, 740 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 28 KiB/s wr, 27 op/s 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/182583550' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:27.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: pgmap v88: 804 pgs: 64 creating+peering, 740 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 28 KiB/s wr, 27 op/s 2024-03-24T11:42:27.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:27.996 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN cmd: Running main() from gmock_main.cc 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [----------] Global test environment set-up. 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (10057 ms) 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (4032 ms) 2024-03-24T11:42:27.997 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-24T11:42:27.998 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (5735 ms) 2024-03-24T11:42:27.998 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [----------] 3 tests from NeoRadosCmd (19824 ms total) 2024-03-24T11:42:27.998 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: 2024-03-24T11:42:27.998 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [----------] Global test environment tear-down 2024-03-24T11:42:27.998 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [==========] 3 tests from 1 test suite ran. (19824 ms total) 2024-03-24T11:42:27.998 INFO:tasks.workunit.client.0.smithi114.stdout: cmd: [ PASSED ] 3 tests. 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:28.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3459601749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.35330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:28.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:28.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:28.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-24T11:42:28.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:28.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:28.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:28.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3459601749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.35330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:28.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:28.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:27 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi114-82433-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi114-83716-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2521153697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3459601749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]: dispatch 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.35330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:28.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:27 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout:: "WaitForSafe_smithi114-82405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.140477+0000 mon.a [INF] from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.140998+0000 mon.a [INF] from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi114-82508-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.141208+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.145064+0000 mon.a [INF] from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi114-83716-1"}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.147928+0000 mon.a [INF] from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi114-82433-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.148175+0000 mon.a [INF] from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:28.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.351042+0000 mon.c [INF] from='client.? v1:172.21.15.114:0/1684261271' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.351179+0000 client.admin [INF] threexx 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.389611+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.390656+0000 mon.a [INF] from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.391353+0000 mon.b [INF] from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.391620+0000 mon.a [INF] from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.787785+0000 mon.a [INF] from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: got: 2024-03-24T11:42:25.792720+0000 mon.a [INF] from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7725 ms) 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (20937 ms total) 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-24T11:42:28.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (20937 ms total) 2024-03-24T11:42:28.979 INFO:tasks.workunit.client.0.smithi114.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-24T11:42:29.073 INFO:tasks.workunit.client.0.smithi114.stdout: list: Running main() from gmock_main.cc 2024-03-24T11:42:29.073 INFO:tasks.workunit.client.0.smithi114.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-24T11:42:29.073 INFO:tasks.workunit.client.0.smithi114.stdout: list: [----------] Global test environment set-up. 2024-03-24T11:42:29.073 INFO:tasks.workunit.client.0.smithi114.stdout: list: [----------] 3 tests from NeoradosList 2024-03-24T11:42:29.073 INFO:tasks.workunit.client.0.smithi114.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [ OK ] NeoradosList.ListObjects (11508 ms) 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [ OK ] NeoradosList.ListObjectsNS (6147 ms) 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3175 ms) 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [----------] 3 tests from NeoradosList (20830 ms total) 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [----------] Global test environment tear-down 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [==========] 3 tests from 1 test suite ran. (20830 ms total) 2024-03-24T11:42:29.074 INFO:tasks.workunit.client.0.smithi114.stdout: list: [ PASSED ] 3 tests. 2024-03-24T11:42:29.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:29.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.35330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2"}]: dispatch 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:29.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: pgmap v91: 764 pgs: 120 unknown, 32 creating+peering, 612 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:29.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:29 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.35330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:29.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: pgmap v91: 764 pgs: 120 unknown, 32 creating+peering, 612 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi114-83598-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi114-82508-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi114-82548-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.34736 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi114-82958-1"}]': finished 2024-03-24T11:42:29.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.35330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-4", "overlaypool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:29.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:29.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:29.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: pgmap v91: 764 pgs: 120 unknown, 32 creating+peering, 612 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:42:29.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:29.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:29.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:29 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: seed 82935 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (106 ms) 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (2 ms) 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-03-24T11:42:30.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1 ms) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (1191 ms) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (1300 ms total) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1202 ms) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (2 ms) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (8 ms) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (8 ms) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1220 ms total) 2024-03-24T11:42:30.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (22060 ms total) 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: Running main() from gmock_main.cc 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [----------] Global test environment set-up. 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (1001 ms) 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: no timeout :/ 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: no timeout :/ 2024-03-24T11:42:30.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: no timeout :/ 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: no timeout :/ 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: no timeout :/ 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (86 ms) 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (15 ms) 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.Checksum (6 ms) 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4 ms) 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (7 ms) 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (8 ms) 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-24T11:42:30.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (5 ms) 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-24T11:42:30.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.RmXattr (14 ms) 2024-03-24T11:42:30.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-24T11:42:30.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-03-24T11:42:30.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [----------] 14 tests from LibRadosIo (1163 ms total) 2024-03-24T11:42:30.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1196 ms) 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (14 ms) 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (9 ms) 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-24T11:42:30.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (9 ms) 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (10 ms) 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1259 ms total) 2024-03-24T11:42:30.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: 2024-03-24T11:42:30.073 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [----------] Global test environment tear-down 2024-03-24T11:42:30.073 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [==========] 24 tests from 2 test suites ran. (22223 ms total) 2024-03-24T11:42:30.073 INFO:tasks.workunit.client.0.smithi114.stdout: api_io: [ PASSED ] 24 tests. 2024-03-24T11:42:30.073 INFO:tasks.workunit.client.0.smithi114.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-24T11:42:30.073 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-24T11:42:30.073 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-24T11:42:30.073 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5 ms) 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (4 ms) 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (2 ms) 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-24T11:42:30.074 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (5 ms) 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (4 ms) 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1292 ms total) 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (22222 ms total) 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: Running main() from gmock_main.cc 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [----------] Global test environment set-up. 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-24T11:42:30.075 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStat.Stat (312 ms) 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (11 ms) 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStat.StatNS (26 ms) 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-03-24T11:42:30.076 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (7 ms) 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [----------] 5 tests from LibRadosStat (357 ms total) 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1169 ms) 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (39 ms) 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1210 ms total) 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: 2024-03-24T11:42:30.077 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [----------] Global test environment tear-down 2024-03-24T11:42:30.078 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (22078 ms total) 2024-03-24T11:42:30.078 INFO:tasks.workunit.client.0.smithi114.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-24T11:42:30.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: pgmap v92: 764 pgs: 120 unknown, 32 creating+peering, 612 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 782 B/s wr, 2 op/s 2024-03-24T11:42:30.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]': finished 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1255912554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]: dispatch 2024-03-24T11:42:30.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:30.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:30 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: pgmap v92: 764 pgs: 120 unknown, 32 creating+peering, 612 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 782 B/s wr, 2 op/s 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]': finished 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1255912554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: pgmap v92: 764 pgs: 120 unknown, 32 creating+peering, 612 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 782 B/s wr, 2 op/s 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi114-83716-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-4-cache", "mode": "writeback"}]': finished 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi114-82603-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1101016855' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2335622928' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:30.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2581088168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1742295566' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1255912554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]: dispatch 2024-03-24T11:42:30.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:30.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:30 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:42:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:31.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3406062976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:31.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/818423872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:31.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:31.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:31.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:42:31.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]: dispatch 2024-03-24T11:42:31.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]: dispatch 2024-03-24T11:42:31.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:31.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3406062976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/818423872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi114-82444-16"}]': finished 2024-03-24T11:42:31.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi114-82459-22"}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi114-82935-7"}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi114-82855-7"}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi114-82405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-4"}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3891146262' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3406062976' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/818423872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:31.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:31.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:31.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:42:31.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:31.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]: dispatch 2024-03-24T11:42:31.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]: dispatch 2024-03-24T11:42:31.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:31.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:42:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: Running main() from gmock_main.cc 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [----------] Global test environment set-up. 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (15121 ms) 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (8715 ms) 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (23836 ms total) 2024-03-24T11:42:32.076 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: 2024-03-24T11:42:32.077 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [----------] Global test environment tear-down 2024-03-24T11:42:32.077 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (23836 ms total) 2024-03-24T11:42:32.077 INFO:tasks.workunit.client.0.smithi114.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-24T11:42:32.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]': finished 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:32.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-24T11:42:32.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]': finished 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:32.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:32.446 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-4", "tierpool": "test-rados-api-smithi114-82677-4-cache"}]': finished 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82958-6", "pg_num": 4}]': finished 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1006945732' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938543332' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:32.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]: dispatch 2024-03-24T11:42:32.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:42:32.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:32.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: pgmap v95: 860 pgs: 248 unknown, 32 creating+peering, 580 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]': finished 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1841779106' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.25390 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:33.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:33.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: pgmap v95: 860 pgs: 248 unknown, 32 creating+peering, 580 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:33.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:33.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:33.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]': finished 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1841779106' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:33.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:33.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:33.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.25390 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:33.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: pgmap v95: 860 pgs: 248 unknown, 32 creating+peering, 580 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi114-82896-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.35291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi114-83598-2"}]': finished 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.25300 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi114-82603-1","app":"app1","key":"key1"}]': finished 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2324752927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1841779106' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.25390 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:33.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: Running main() from gmock_main.cc 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [----------] Global test environment set-up. 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (308 ms) 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.LockShared (11 ms) 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1009 ms) 2024-03-24T11:42:34.192 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1008 ms) 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5 ms) 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [----------] 8 tests from LibRadosLock (2352 ms total) 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1152 ms) 2024-03-24T11:42:34.193 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-24T11:42:34.194 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6 ms) 2024-03-24T11:42:34.194 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-24T11:42:34.194 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1006 ms) 2024-03-24T11:42:34.194 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-24T11:42:34.194 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1010 ms) 2024-03-24T11:42:34.194 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (5 ms) 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (5 ms) 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (2 ms) 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3191 ms total) 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [----------] Global test environment tear-down 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (26328 ms total) 2024-03-24T11:42:34.195 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: seed 82548 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (309 ms) 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (8 ms) 2024-03-24T11:42:34.210 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1020 ms) 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1024 ms) 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3 ms) 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (3 ms) 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (2 ms) 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2373 ms total) 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-24T11:42:34.211 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1157 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (4 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1079 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1004 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-03-24T11:42:34.212 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3258 ms total) 2024-03-24T11:42:34.213 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: 2024-03-24T11:42:34.213 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-24T11:42:34.213 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (26317 ms total) 2024-03-24T11:42:34.213 INFO:tasks.workunit.client.0.smithi114.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-24T11:42:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: pgmap v98: 720 pgs: 4 creating+peering, 104 unknown, 612 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.0 KiB/s wr, 4 op/s 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]': finished 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.25390 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]: dispatch 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1655387886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2991414653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]: dispatch 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]': finished 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:42:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:34.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:34.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2537786140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:34.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.26317 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:34 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:34.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: pgmap v98: 720 pgs: 4 creating+peering, 104 unknown, 612 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.0 KiB/s wr, 4 op/s 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]': finished 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.25390 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]: dispatch 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1655387886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2991414653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:34.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]: dispatch 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]': finished 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:34.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:34.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2537786140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:34.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.26317 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:34.657 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:42:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:34.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: pgmap v98: 720 pgs: 4 creating+peering, 104 unknown, 612 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.0 KiB/s wr, 4 op/s 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi114-83716-2"}]': finished 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "overlaypool": "test-rados-api-smithi114-82958-6"}]': finished 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.25390 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi114-82405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/355770799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3235162993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]: dispatch 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1655387886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2991414653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]: dispatch 2024-03-24T11:42:34.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]: dispatch 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]: dispatch 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi114-82548-10"}]': finished 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi114-82508-10"}]': finished 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82958-6", "mode": "writeback"}]': finished 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi114-83716-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:34.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:34.660 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2537786140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.660 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:34.660 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.26317 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:34.660 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:34 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:35.773 INFO:tasks.workunit.client.0.smithi114.stdout: pool: Running main() from gmock_main.cc 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [----------] Global test environment set-up. 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ OK ] NeoRadosPools.PoolList (9980 ms) 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4043 ms) 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3624 ms) 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5440 ms) 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-24T11:42:35.774 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1839 ms) 2024-03-24T11:42:35.775 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-24T11:42:35.775 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2596 ms) 2024-03-24T11:42:35.775 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [----------] 6 tests from NeoRadosPools (27522 ms total) 2024-03-24T11:42:35.775 INFO:tasks.workunit.client.0.smithi114.stdout: pool: 2024-03-24T11:42:35.775 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [----------] Global test environment tear-down 2024-03-24T11:42:35.775 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [==========] 6 tests from 1 test suite ran. (27522 ms total) 2024-03-24T11:42:35.775 INFO:tasks.workunit.client.0.smithi114.stdout: pool: [ PASSED ] 6 tests. 2024-03-24T11:42:36.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:36.515 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-24T11:42:36.515 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify_test_cb 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (338 ms) 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94784955749664 err -107 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (13 ms) 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94784955751712 err -107 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1008 ms) 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-24T11:42:36.516 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 335007449090 cookie 94784955757760 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (4 ms) 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 335007449089 cookie 94784955757760 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (4 ms) 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 335007449088 cookie 94784955799344 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (4 ms) 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 335007449088 cookie 94784955778864 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 335007449088 cookie 94784955817056 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (6 ms) 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 335007449091 cookie 94784955778864 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 339302416388 cookie 94784955778864 2024-03-24T11:42:36.517 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3004 ms) 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94784955840416 err -107 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_cb from 24745 notify_id 365072220161 cookie 94784955840416 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5064 ms) 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94784955840416 err -107 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1031 ms) 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10476 ms total) 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: watch_notify_test_cb 2024-03-24T11:42:36.518 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1308 ms) 2024-03-24T11:42:36.519 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1308 ms total) 2024-03-24T11:42:36.519 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: 2024-03-24T11:42:36.519 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-24T11:42:36.519 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (28521 ms total) 2024-03-24T11:42:36.519 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-24T11:42:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: pgmap v101: 680 pgs: 4 creating+peering, 160 unknown, 516 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.7 KiB/s wr, 3 op/s 2024-03-24T11:42:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.26317 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2494194617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.27364 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.27364 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2098249788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[30312]: from='client.29137 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: pgmap v101: 680 pgs: 4 creating+peering, 160 unknown, 516 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.7 KiB/s wr, 3 op/s 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.26317 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2494194617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.27364 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.27364 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2098249788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:36 smithi114 ceph-mon[42466]: from='client.29137 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: pgmap v101: 680 pgs: 4 creating+peering, 160 unknown, 516 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.7 KiB/s wr, 3 op/s 2024-03-24T11:42:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.26317 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi114-82736-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1749471152' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2494194617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.27364 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi114-83716-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.35366 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi114-82896-12"}]': finished 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.27364 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi114-82405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:37.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:42:37.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2098249788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:36 smithi119 ceph-mon[33714]: from='client.29137 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: notify 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2714 ms) 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (2719 ms total) 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: 2024-03-24T11:42:37.606 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: notify 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (80 ms) 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: notify 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4209 ms) 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (95 ms) 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (46 ms) 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519622608 notify_id 395136991234 notifier_gid 35345 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (18 ms) 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519622608 notify_id 395136991233 notifier_gid 35345 2024-03-24T11:42:37.607 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (19 ms) 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519622608 notify_id 395136991233 notifier_gid 35345 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (16 ms) 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519622608 notify_id 395136991235 notifier_gid 35345 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (11 ms) 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519622608 notify_id 395136991236 notifier_gid 35345 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (14 ms) 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519622528 notify_id 395136991237 notifier_gid 35345 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (14 ms) 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: trying... 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519622528 notify_id 395136991238 notifier_gid 35345 2024-03-24T11:42:37.608 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: timed out 2024-03-24T11:42:37.609 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: flushing 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: pgmap v104: 720 pgs: 96 creating+peering, 72 unknown, 552 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 6.2 KiB/s wr, 4 op/s 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: pool 'PoolQuotaPP_smithi114-82433-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: from='client.29137 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2773582056' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: from='client.30040 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: pgmap v104: 720 pgs: 96 creating+peering, 72 unknown, 552 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 6.2 KiB/s wr, 4 op/s 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: pool 'PoolQuotaPP_smithi114-82433-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: from='client.29137 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2773582056' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: from='client.30040 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: pgmap v104: 720 pgs: 96 creating+peering, 72 unknown, 552 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 6.2 KiB/s wr, 4 op/s 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: pool 'PoolQuotaPP_smithi114-82433-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: from='client.29137 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2773582056' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: from='client.30040 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:39.395 INFO:tasks.workunit.client.0.smithi114.stdout: a io: 2024-03-24T11:42:39.395 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-24T11:42:39.395 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.395 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: cluster: 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: id: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: health: HEALTH_WARN 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 20 pool(s) do not have an application enabled 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: services: 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: data: 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pools: 26 pools, 772 pgs 2024-03-24T11:42:39.396 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: pgs: 21.891% pgs unknown 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 58.161% pgs not active 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 449 creating+peering 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 169 unknown 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 154 active+clean 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: io: 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: client: 1.3 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6193 ms) 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ OK ] LibRadosService.Status (20045 ms) 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [----------] 4 tests from LibRadosService (31342 ms total) 2024-03-24T11:42:39.397 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: 2024-03-24T11:42:39.398 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [----------] Global test environment tear-down 2024-03-24T11:42:39.398 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31342 ms total) 2024-03-24T11:42:39.398 INFO:tasks.workunit.client.0.smithi114.stdout: api_service: [ PASSED ] 4 tests. 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5051 ms) 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (45 ms) 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (21034 ms) 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: attempt 0 of 20 2024-03-24T11:42:39.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5255 ms) 2024-03-24T11:42:39.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31385 ms total) 2024-03-24T11:42:39.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: 2024-03-24T11:42:39.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-24T11:42:39.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31386 ms total) 2024-03-24T11:42:39.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-24T11:42:39.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:39.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[42466]: from='client.30040 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:39.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[42466]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:42:39.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:39.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:39.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:39.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[30312]: from='client.30040 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:39.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[30312]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:42:39.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:39.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:39 smithi119 ceph-mon[33714]: from='client.30040 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:39 smithi119 ceph-mon[33714]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:42:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:40.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: pgmap v107: 624 pgs: 32 creating+peering, 72 unknown, 520 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 6.2 KiB/s wr, 5 op/s 2024-03-24T11:42:40.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/393357960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1338367635' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[30312]: from='client.32002 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: pgmap v107: 624 pgs: 32 creating+peering, 72 unknown, 520 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 6.2 KiB/s wr, 5 op/s 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/393357960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:40.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1338367635' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-24T11:42:40.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-24T11:42:40.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:40.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:42:40.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:40 smithi114 ceph-mon[42466]: from='client.32002 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: pgmap v107: 624 pgs: 32 creating+peering, 72 unknown, 520 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 6.2 KiB/s wr, 5 op/s 2024-03-24T11:42:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/393357960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1338367635' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-24T11:42:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-24T11:42:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:42:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:40 smithi119 ceph-mon[33714]: from='client.32002 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.32002 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:42:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3400239647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.32008 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.32008 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]': finished 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2604975230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/687442104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[30312]: from='client.42803 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.32002 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3400239647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.32008 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:41.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.32008 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]': finished 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2604975230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/687442104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.743 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:41 smithi114 ceph-mon[42466]: from='client.42803 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:42:41.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.32002 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi114-82839-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3400239647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.32008 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.32008 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi114-82405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-7"}]': finished 2024-03-24T11:42:41.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:42:41.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2201983843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2604975230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/687442104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:41.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]: dispatch 2024-03-24T11:42:41.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:41 smithi119 ceph-mon[33714]: from='client.42803 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:42.452 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: Running main() from gmock_main.cc 2024-03-24T11:42:42.452 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [----------] Global test environment set-up. 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (17627 ms) 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7283 ms) 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (9288 ms) 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [----------] 3 tests from NeoradosECList (34198 ms total) 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [----------] Global test environment tear-down 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (34198 ms total) 2024-03-24T11:42:42.453 INFO:tasks.workunit.client.0.smithi114.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-24T11:42:43.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: pgmap v110: 712 pgs: 32 creating+peering, 160 unknown, 520 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/687442104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.42803 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:43.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:43.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: pgmap v110: 712 pgs: 32 creating+peering, 160 unknown, 520 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:43.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:43.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:43.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:43.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/687442104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:43.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:43.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.42803 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:43.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:42:43.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:42:43.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:43.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:43.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: pgmap v110: 712 pgs: 32 creating+peering, 160 unknown, 520 active+clean; 145 MiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/687442104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-83146-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi114-83716-3"}]': finished 2024-03-24T11:42:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.42803 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:42:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:42:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]: dispatch 2024-03-24T11:42:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: pgmap v113: 712 pgs: 96 unknown, 616 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]': finished 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3206267734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:44.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[30312]: from='client.45041 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: pgmap v113: 712 pgs: 96 unknown, 616 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]': finished 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3206267734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:44.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:43 smithi114 ceph-mon[42466]: from='client.45041 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: pgmap v113: 712 pgs: 96 unknown, 616 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-7"}]': finished 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:42:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3206267734' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:43 smithi119 ceph-mon[33714]: from='client.45041 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:44.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:42:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:42:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:45 smithi114 ceph-mon[30312]: from='client.45041 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:45 smithi114 ceph-mon[30312]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:42:45.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:45.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:45 smithi114 ceph-mon[42466]: from='client.45041 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:45.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:45 smithi114 ceph-mon[42466]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:42:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:45 smithi119 ceph-mon[33714]: from='client.45041 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi114-82405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:45 smithi119 ceph-mon[33714]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:42:46.912 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-24T11:42:46.912 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-24T11:42:46.912 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-24T11:42:46.912 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-24T11:42:46.912 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (11671 ms) 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (6146 ms) 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3176 ms) 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3024 ms) 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4422 ms) 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-24T11:42:46.913 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3945 ms) 2024-03-24T11:42:46.914 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-24T11:42:46.914 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (6454 ms) 2024-03-24T11:42:46.914 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (38839 ms total) 2024-03-24T11:42:46.914 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: 2024-03-24T11:42:46.914 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-24T11:42:46.914 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (38839 ms total) 2024-03-24T11:42:46.914 INFO:tasks.workunit.client.0.smithi114.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[30312]: pgmap v116: 648 pgs: 128 unknown, 520 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[30312]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3647339448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[30312]: from='client.39139 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:46.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[42466]: pgmap v116: 648 pgs: 128 unknown, 520 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:42:46.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:46.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[42466]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:42:46.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3647339448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:46.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[42466]: from='client.39139 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:46.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:46.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:46 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:46 smithi119 ceph-mon[33714]: pgmap v116: 648 pgs: 128 unknown, 520 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:42:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:46 smithi119 ceph-mon[33714]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:42:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3647339448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:46 smithi119 ceph-mon[33714]: from='client.39139 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:46 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:47.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]: dispatch 2024-03-24T11:42:47.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]: dispatch 2024-03-24T11:42:47.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: pgmap v118: 680 pgs: 96 creating+peering, 584 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.39139 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]': finished 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3515230967' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.40423 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1033459800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.48110 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:47.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.40423 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]': finished 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.48110 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]: dispatch 2024-03-24T11:42:47.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: pgmap v118: 680 pgs: 96 creating+peering, 584 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.39139 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]': finished 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3515230967' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.40423 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1033459800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.48110 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.40423 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]': finished 2024-03-24T11:42:47.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.48110 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:47.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:42:47.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:42:47.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:47.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:47 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:48.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:48.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]: dispatch 2024-03-24T11:42:48.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: pgmap v118: 680 pgs: 96 creating+peering, 584 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.39139 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4"}]': finished 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3515230967' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.40423 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1033459800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.48110 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.40423 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi114-82405-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:48.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi114-82958-4", "tierpool": "test-rados-api-smithi114-82958-6"}]': finished 2024-03-24T11:42:48.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.48110 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:48.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:42:48.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1695270246' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:42:48.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:48.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:47 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:42:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:42:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:42:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:49.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:42:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi114-82958-6", "pool2": "test-rados-api-smithi114-82958-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:42:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:49.472 INFO:tasks.workunit.client.0.smithi114.stdout:pi_watch_notify_pp: flushed 2024-03-24T11:42:49.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3022 ms) 2024-03-24T11:42:49.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-24T11:42:49.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: trying... 2024-03-24T11:42:49.472 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519620480 notify_id 408021893127 notifier_gid 35345 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: timed out 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: flushing 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: flushed 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3010 ms) 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: List watches 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: notify2 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519620480 notify_id 420906795016 notifier_gid 35345 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: notify2 done 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: watch_check 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: unwatch2 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: flushing 2024-03-24T11:42:49.473 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: done 2024-03-24T11:42:49.474 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3121 ms) 2024-03-24T11:42:49.474 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-24T11:42:49.474 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: List watches 2024-03-24T11:42:49.474 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: notify2 2024-03-24T11:42:49.474 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: handle_notify cookie 94452519620480 notify_id 433791696905 notifier_gid 35345 2024-03-24T11:42:49.474 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: notify2 done 2024-03-24T11:42:49.474 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: watch_check 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: unwatch2 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: flushing 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: done 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3025 ms) 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16700 ms total) 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-24T11:42:49.475 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (41466 ms total) 2024-03-24T11:42:49.476 INFO:tasks.workunit.client.0.smithi114.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-24T11:42:50.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: pgmap v122: 644 pgs: 96 unknown, 32 creating+peering, 516 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]': finished 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2158733078' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.50660 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/832955542' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.43141 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:50.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: pgmap v122: 644 pgs: 96 unknown, 32 creating+peering, 516 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]': finished 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]: dispatch 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]: dispatch 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2158733078' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.50660 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/832955542' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.43141 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: pgmap v122: 644 pgs: 96 unknown, 32 creating+peering, 516 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-9"}]': finished 2024-03-24T11:42:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]: dispatch 2024-03-24T11:42:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:42:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]: dispatch 2024-03-24T11:42:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2158733078' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.50660 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/832955542' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.43141 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]': finished 2024-03-24T11:42:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-mon[33714]: from='client.50660 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-mon[33714]: from='client.43141 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-mon[33714]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:42:51.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]': finished 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[30312]: from='client.50660 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[30312]: from='client.43141 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[30312]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-9", "mode": "writeback"}]': finished 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[42466]: from='client.50660 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi114-82405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[42466]: from='client.43141 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi114-82433-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[42466]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:42:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:52.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:42:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:42:52.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:52.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: pgmap v125: 644 pgs: 128 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: 22.1 deep-scrub starts 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: 22.1 deep-scrub ok 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:52.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: 22.3 deep-scrub starts 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:52.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: pgmap v125: 644 pgs: 128 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: 22.1 deep-scrub starts 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: 22.1 deep-scrub ok 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: 22.3 deep-scrub starts 2024-03-24T11:42:52.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: 22.3 deep-scrub ok 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: 22.7 deep-scrub starts 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: 22.7 deep-scrub ok 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: 22.3 deep-scrub ok 2024-03-24T11:42:52.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: 22.7 deep-scrub starts 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: 22.7 deep-scrub ok 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:52.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.0"}]: dispatch 2024-03-24T11:42:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.1"}]: dispatch 2024-03-24T11:42:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.2"}]: dispatch 2024-03-24T11:42:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.3"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.4"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.5"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.6"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.7"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.8"}]: dispatch 2024-03-24T11:42:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "22.9"}]: dispatch 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: pgmap v125: 644 pgs: 128 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 790 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: 22.1 deep-scrub starts 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: 22.1 deep-scrub ok 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: 22.3 deep-scrub starts 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: 22.3 deep-scrub ok 2024-03-24T11:42:53.057 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: 22.7 deep-scrub starts 2024-03-24T11:42:53.058 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:42:53.058 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: 22.7 deep-scrub ok 2024-03-24T11:42:53.058 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:42:53.058 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:53.058 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]: dispatch 2024-03-24T11:42:53.058 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.059 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.059 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:53.711 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] Global test environment set-up. 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: 2024-03-24T11:42:07.954+0000 7f883fc38780 -1 failed for service _ceph-mon._tcp 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: 2024-03-24T11:42:07.954+0000 7f883fc38780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (48 ms) 2024-03-24T11:42:53.712 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2011 ms) 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2059 ms total) 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: started 0x55e36e476600 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: started 0x55e36e843a40 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: started 2 aios 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: waiting 0x55e36e476600 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: waiting 0x55e36e843a40 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: done. 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (14401 ms) 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (14401 ms total) 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: 2024-03-24T11:42:53.713 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.Exec (61 ms) 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (5 ms) 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2 ms) 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5030 ms) 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-24T11:42:53.714 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-24T11:42:53.715 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-24T11:42:53.715 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (20813 ms) 2024-03-24T11:42:53.715 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] 8 tests from LibRadosMisc (25911 ms total) 2024-03-24T11:42:53.715 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: 2024-03-24T11:42:53.715 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [----------] Global test environment tear-down 2024-03-24T11:42:53.715 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (45795 ms total) 2024-03-24T11:42:53.715 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-24T11:42:53.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.8 deep-scrub starts 2024-03-24T11:42:53.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.8 deep-scrub ok 2024-03-24T11:42:53.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.6 deep-scrub starts 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.6 deep-scrub ok 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.5 deep-scrub starts 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.5 deep-scrub ok 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.0 deep-scrub starts 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.0 deep-scrub ok 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.4 deep-scrub starts 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: 22.4 deep-scrub ok 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]': finished 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:42:53.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4063355960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.46933 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/143700869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.46945 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: pgmap v128: 612 pgs: 128 unknown, 484 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:53.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.8 deep-scrub starts 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.8 deep-scrub ok 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.6 deep-scrub starts 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.6 deep-scrub ok 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.5 deep-scrub starts 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.5 deep-scrub ok 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.0 deep-scrub starts 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.0 deep-scrub ok 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.4 deep-scrub starts 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: 22.4 deep-scrub ok 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]': finished 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:42:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4063355960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.46933 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/143700869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.46945 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: pgmap v128: 612 pgs: 128 unknown, 484 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:53.993 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:54.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.8 deep-scrub starts 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.8 deep-scrub ok 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.6 deep-scrub starts 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.6 deep-scrub ok 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.5 deep-scrub starts 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.5 deep-scrub ok 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.0 deep-scrub starts 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.0 deep-scrub ok 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.4 deep-scrub starts 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: 22.4 deep-scrub ok 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-9"}]': finished 2024-03-24T11:42:54.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82730-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4063355960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.46933 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/143700869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.46945 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: pgmap v128: 612 pgs: 128 unknown, 484 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:42:54.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:54.969 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:42:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.46933 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.46945 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]: dispatch 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]: dispatch 2024-03-24T11:42:55.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.46933 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.46945 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]: dispatch 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]: dispatch 2024-03-24T11:42:55.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.46933 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi114-82405-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.46945 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi114-82433-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:42:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:42:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:42:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]: dispatch 2024-03-24T11:42:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]: dispatch 2024-03-24T11:42:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]': finished 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: pgmap v131: 516 pgs: 96 unknown, 420 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 2 op/s 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]: dispatch 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/346362577' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]: dispatch 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[30312]: from='client.48490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]': finished 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: pgmap v131: 516 pgs: 96 unknown, 420 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 2 op/s 2024-03-24T11:42:56.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]: dispatch 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/346362577' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]: dispatch 2024-03-24T11:42:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:55 smithi114 ceph-mon[42466]: from='client.48490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache", "force_nonempty":""}]': finished 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: pgmap v131: 516 pgs: 96 unknown, 420 active+clean; 217 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 2 op/s 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi114-82736-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82730-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3226035701' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]: dispatch 2024-03-24T11:42:56.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/346362577' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:56.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:42:56.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]: dispatch 2024-03-24T11:42:56.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:55 smithi119 ceph-mon[33714]: from='client.48490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]': finished 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: from='client.48490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3175287886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: from='client.48493 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]': finished 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: from='client.48490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3175287886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: from='client.48493 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:56 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: from='client.45649 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82730-10", "tierpool":"test-rados-api-smithi114-82730-10-cache"}]': finished 2024-03-24T11:42:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: from='client.48490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi114-82433-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:57.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:42:57.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3175287886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:57.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: from='client.48493 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:57.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:57.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:56 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:42:57.731 INFO:tasks.workunit.client.0.smithi114.stdout: misc: Running main() from gmock_main.cc 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [----------] Global test environment set-up. 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.Version (9682 ms) 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4330 ms) 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-24T11:42:57.732 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.LongName (5738 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3336 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2864 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4083 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.Exec (4156 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (4434 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2601 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3241 ms) 2024-03-24T11:42:57.733 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2248 ms) 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2767 ms) 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [----------] 12 tests from NeoRadosMisc (49480 ms total) 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [----------] Global test environment tear-down 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [==========] 12 tests from 1 test suite ran. (49481 ms total) 2024-03-24T11:42:57.734 INFO:tasks.workunit.client.0.smithi114.stdout: misc: [ PASSED ] 12 tests. 2024-03-24T11:42:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: 22.2 deep-scrub starts 2024-03-24T11:42:58.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: 22.2 deep-scrub ok 2024-03-24T11:42:58.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: pgmap v134: 652 pgs: 32 unknown, 104 creating+peering, 516 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-24T11:42:58.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:58.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: from='client.48493 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:58.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:58.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:42:58.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:42:58.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:42:58.724 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: Running main() from gmock_main.cc 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [----------] Global test environment set-up. 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolList (10682 ms) 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5755 ms) 2024-03-24T11:42:58.725 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3637 ms) 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4108 ms) 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3661 ms) 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4678 ms) 2024-03-24T11:42:58.726 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5412 ms) 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4644 ms) 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8214 ms) 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [----------] 9 tests from LibRadosPools (50791 ms total) 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: 2024-03-24T11:42:58.727 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [----------] Global test environment tear-down 2024-03-24T11:42:58.728 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (50791 ms total) 2024-03-24T11:42:58.728 INFO:tasks.workunit.client.0.smithi114.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-24T11:42:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: 22.2 deep-scrub starts 2024-03-24T11:42:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: 22.2 deep-scrub ok 2024-03-24T11:42:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: pgmap v134: 652 pgs: 32 unknown, 104 creating+peering, 516 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-24T11:42:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:58.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: from='client.48493 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:58.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:58.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:42:58.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:42:58.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:42:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: 22.2 deep-scrub starts 2024-03-24T11:42:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: 22.2 deep-scrub ok 2024-03-24T11:42:58.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: pgmap v134: 652 pgs: 32 unknown, 104 creating+peering, 516 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-24T11:42:58.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:58.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: from='client.48493 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi114-82405-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:42:58.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:42:58.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:42:58.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:42:58.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:42:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: 22.9 deep-scrub starts 2024-03-24T11:42:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: 22.9 deep-scrub ok 2024-03-24T11:42:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]': finished 2024-03-24T11:42:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:42:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]: dispatch 2024-03-24T11:42:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]: dispatch 2024-03-24T11:42:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1814255778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:42:59 smithi119 ceph-mon[33714]: from='client.55472 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: 22.9 deep-scrub starts 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: 22.9 deep-scrub ok 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]': finished 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]: dispatch 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]: dispatch 2024-03-24T11:42:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1814255778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[30312]: from='client.55472 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: 22.9 deep-scrub starts 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: 22.9 deep-scrub ok 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-11"}]': finished 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]: dispatch 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]: dispatch 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1814255778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:42:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:42:59 smithi114 ceph-mon[42466]: from='client.55472 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: pgmap v137: 524 pgs: 64 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 4 op/s 2024-03-24T11:43:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:00.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]': finished 2024-03-24T11:43:00.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: from='client.55472 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:00.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:43:00.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/800270789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:00.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:00 smithi119 ceph-mon[33714]: from='client.55478 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: pgmap v137: 524 pgs: 64 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 4 op/s 2024-03-24T11:43:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]': finished 2024-03-24T11:43:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: from='client.55472 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/800270789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[30312]: from='client.55478 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: pgmap v137: 524 pgs: 64 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 4 op/s 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-11", "mode": "writeback"}]': finished 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: from='client.55472 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi114-82433-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:43:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/800270789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:00 smithi114 ceph-mon[42466]: from='client.55478 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:01 smithi119 ceph-mon[33714]: from='client.55478 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:01 smithi119 ceph-mon[33714]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:43:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[30312]: from='client.55478 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[30312]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[42466]: from='client.55478 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi114-82405-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[42466]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:43:01.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:02.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:43:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:43:02.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: pgmap v140: 492 pgs: 32 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:02.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:02.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:43:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:43:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:43:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4185229324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:02 smithi119 ceph-mon[33714]: from='client.55490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: pgmap v140: 492 pgs: 32 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4185229324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:02.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[30312]: from='client.55490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: pgmap v140: 492 pgs: 32 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 982 MiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]: dispatch 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4185229324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:02.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:02 smithi114 ceph-mon[42466]: from='client.55490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:03.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]': finished 2024-03-24T11:43:03.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: from='client.55490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:03.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:43:03.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/425518808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:03.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:03 smithi119 ceph-mon[33714]: from='client.48526 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:03.697 INFO:tasks.workunit.client.0.smithi114.stdout: io: Running main() from gmock_main.cc 2024-03-24T11:43:03.697 INFO:tasks.workunit.client.0.smithi114.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [----------] Global test environment set-up. 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.Limits (11543 ms) 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (6143 ms) 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.ReadOp (3184 ms) 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.SparseRead (3017 ms) 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3669 ms) 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (2513 ms) 2024-03-24T11:43:03.698 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3163 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3208 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3704 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.Trunc (3173 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.Remove (3448 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2763 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.RmXattr (3007 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2962 ms) 2024-03-24T11:43:03.699 INFO:tasks.workunit.client.0.smithi114.stdout: io: [----------] 14 tests from NeoRadosIo (55497 ms total) 2024-03-24T11:43:03.700 INFO:tasks.workunit.client.0.smithi114.stdout: io: 2024-03-24T11:43:03.700 INFO:tasks.workunit.client.0.smithi114.stdout: io: [----------] Global test environment tear-down 2024-03-24T11:43:03.700 INFO:tasks.workunit.client.0.smithi114.stdout: io: [==========] 14 tests from 1 test suite ran. (55497 ms total) 2024-03-24T11:43:03.700 INFO:tasks.workunit.client.0.smithi114.stdout: io: [ PASSED ] 14 tests. 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]': finished 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: from='client.55490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:43:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/425518808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[30312]: from='client.48526 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-11"}]': finished 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: from='client.55490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/425518808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:03 smithi114 ceph-mon[42466]: from='client.48526 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:04 smithi119 ceph-mon[33714]: pgmap v143: 556 pgs: 96 unknown, 460 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 258 KiB/s wr, 7 op/s 2024-03-24T11:43:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:04 smithi119 ceph-mon[33714]: from='client.48526 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:04 smithi119 ceph-mon[33714]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[30312]: pgmap v143: 556 pgs: 96 unknown, 460 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 258 KiB/s wr, 7 op/s 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[30312]: from='client.48526 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[30312]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[42466]: pgmap v143: 556 pgs: 96 unknown, 460 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 258 KiB/s wr, 7 op/s 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[42466]: from='client.48526 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi114-82405-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-mon[42466]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:43:04.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:43:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:43:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:05.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:05.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:05 smithi119 ceph-mon[33714]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:43:05.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4232588398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:05 smithi119 ceph-mon[33714]: from='client.48532 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[30312]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:43:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4232588398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[30312]: from='client.48532 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:05.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[42466]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:43:05.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4232588398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:05.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:05 smithi114 ceph-mon[42466]: from='client.48532 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: pgmap v146: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-24T11:43:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: from='client.48532 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:43:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2158826927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi114-82405-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:06 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: pgmap v146: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: from='client.48532 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2158826927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi114-82405-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: pgmap v146: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: from='client.48532 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi114-82433-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2158826927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi114-82405-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:06 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:07.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:07.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2158826927' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi114-82405-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:07.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:07 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:43:07.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:07.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:07 smithi119 ceph-mon[33714]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:43:07.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:07 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2158826927' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi114-82405-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[30312]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2158826927' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi114-82405-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:07.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:43:07.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1357698478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:07.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[42466]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:43:07.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:07 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]: dispatch 2024-03-24T11:43:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: pgmap v149: 484 pgs: 32 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 24 op/s 2024-03-24T11:43:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2988883930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.48544 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:08.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:08 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: pgmap v149: 484 pgs: 32 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 24 op/s 2024-03-24T11:43:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:43:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:43:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2988883930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.48544 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: pgmap v149: 484 pgs: 32 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 24 op/s 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.48403 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi114-82736-10"}]': finished 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:43:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2988883930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:08.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.48544 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:08.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:08.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:08.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:08 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: from='client.48544 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2358036217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:09 smithi119 ceph-mon[33714]: from='client.48553 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: from='client.48544 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2358036217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[30312]: from='client.48553 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: from='client.48544 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi114-82433-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2358036217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:09.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:09 smithi114 ceph-mon[42466]: from='client.48553 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:10 smithi119 ceph-mon[33714]: pgmap v152: 516 pgs: 64 unknown, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T11:43:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:10 smithi119 ceph-mon[33714]: from='client.48553 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:10.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:10 smithi119 ceph-mon[33714]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:43:10.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:10.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[30312]: pgmap v152: 516 pgs: 64 unknown, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T11:43:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[30312]: from='client.48553 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[30312]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:43:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:10.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[42466]: pgmap v152: 516 pgs: 64 unknown, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-24T11:43:10.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:10.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[42466]: from='client.48553 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi114-82405-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:10.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[42466]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:43:10.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:10.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:43:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-mon[33714]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:43:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1737723690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi114-82433-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:11.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:43:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[30312]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1737723690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi114-82433-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi114-82736-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[42466]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1737723690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi114-82433-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:11.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:12.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:43:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:43:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: pgmap v155: 492 pgs: 40 unknown, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:12.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1737723690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi114-82433-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:12.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]': finished 2024-03-24T11:43:12.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]: dispatch 2024-03-24T11:43:12.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:43:12.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]: dispatch 2024-03-24T11:43:12.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1327498666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi114-82405-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: pgmap v155: 492 pgs: 40 unknown, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1737723690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi114-82433-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]': finished 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]: dispatch 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]: dispatch 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1327498666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi114-82405-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: pgmap v155: 492 pgs: 40 unknown, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:12.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:12.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1737723690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi114-82433-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:12.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-13"}]': finished 2024-03-24T11:43:12.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]: dispatch 2024-03-24T11:43:12.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:43:12.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]: dispatch 2024-03-24T11:43:12.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1327498666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi114-82405-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:13 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:13 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:13 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]': finished 2024-03-24T11:43:13.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1327498666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi114-82405-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:13.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:13 smithi119 ceph-mon[33714]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]': finished 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1327498666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi114-82405-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[42466]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:13.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:13.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:13.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:13.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-13", "mode": "writeback"}]': finished 2024-03-24T11:43:13.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1327498666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi114-82405-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:13.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:13 smithi114 ceph-mon[30312]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: pgmap v158: 492 pgs: 32 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 KiB/s wr, 101 op/s 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.0"}]: dispatch 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.0"}]: dispatch 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.1"}]: dispatch 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.1"}]: dispatch 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.2"}]: dispatch 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.2"}]: dispatch 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.3"}]: dispatch 2024-03-24T11:43:14.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.3"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.4"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.4"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.5"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.5"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.6"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.6"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.7"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.7"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.8"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.8"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.9"}]: dispatch 2024-03-24T11:43:14.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.9"}]: dispatch 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2877002244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: from='client.48568 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: 145.0 scrub starts 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: 145.0 scrub ok 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: 145.9 scrub starts 2024-03-24T11:43:14.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:14 smithi119 ceph-mon[33714]: 145.9 scrub ok 2024-03-24T11:43:14.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: pgmap v158: 492 pgs: 32 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 KiB/s wr, 101 op/s 2024-03-24T11:43:14.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.0"}]: dispatch 2024-03-24T11:43:14.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.0"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.1"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.1"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.2"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.2"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.3"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.3"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.4"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.4"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.5"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.5"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.6"}]: dispatch 2024-03-24T11:43:14.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.6"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.7"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.7"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.8"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.8"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.9"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.9"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:43:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2877002244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: pgmap v158: 492 pgs: 32 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 KiB/s wr, 101 op/s 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.0"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.0"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.1"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.1"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.2"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.2"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.3"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.3"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.4"}]: dispatch 2024-03-24T11:43:14.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.4"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.5"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.5"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.6"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.6"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.7"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.7"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.8"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.8"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "145.9"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "145.9"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:14.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2877002244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: from='client.48568 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: 145.0 scrub starts 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: 145.0 scrub ok 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: 145.9 scrub starts 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[30312]: 145.9 scrub ok 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: from='client.48568 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: 145.0 scrub starts 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: 145.0 scrub ok 2024-03-24T11:43:14.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: 145.9 scrub starts 2024-03-24T11:43:14.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-mon[42466]: 145.9 scrub ok 2024-03-24T11:43:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:43:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:43:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.5 scrub starts 2024-03-24T11:43:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.5 scrub ok 2024-03-24T11:43:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.2 scrub starts 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.2 scrub ok 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.3 scrub starts 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.3 scrub ok 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.7 scrub starts 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.7 scrub ok 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.8 scrub starts 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.8 scrub ok 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: from='client.48568 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1489401826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi114-82405-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.4 scrub starts 2024-03-24T11:43:15.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:15 smithi119 ceph-mon[33714]: 145.4 scrub ok 2024-03-24T11:43:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.5 scrub starts 2024-03-24T11:43:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.5 scrub ok 2024-03-24T11:43:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.2 scrub starts 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.2 scrub ok 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.3 scrub starts 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.3 scrub ok 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.7 scrub starts 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.7 scrub ok 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.8 scrub starts 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.8 scrub ok 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: from='client.48568 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1489401826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi114-82405-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:15.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.4 scrub starts 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[30312]: 145.4 scrub ok 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.5 scrub starts 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.5 scrub ok 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.2 scrub starts 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.2 scrub ok 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.3 scrub starts 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.3 scrub ok 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.7 scrub starts 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.7 scrub ok 2024-03-24T11:43:15.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.8 scrub starts 2024-03-24T11:43:15.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.8 scrub ok 2024-03-24T11:43:15.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:15.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: from='client.48568 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi114-82433-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:15.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:43:15.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1489401826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi114-82405-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:15.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.4 scrub starts 2024-03-24T11:43:15.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:15 smithi114 ceph-mon[42466]: 145.4 scrub ok 2024-03-24T11:43:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: pgmap v161: 524 pgs: 64 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 KiB/s wr, 101 op/s 2024-03-24T11:43:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: 145.6 scrub starts 2024-03-24T11:43:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: 145.6 scrub ok 2024-03-24T11:43:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: 145.1 scrub starts 2024-03-24T11:43:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: 145.1 scrub ok 2024-03-24T11:43:16.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:16.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1489401826' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi114-82405-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:16.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:16 smithi119 ceph-mon[33714]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: pgmap v161: 524 pgs: 64 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 KiB/s wr, 101 op/s 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: 145.6 scrub starts 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: 145.6 scrub ok 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: 145.1 scrub starts 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: 145.1 scrub ok 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1489401826' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi114-82405-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[30312]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: pgmap v161: 524 pgs: 64 unknown, 8 creating+peering, 452 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 KiB/s wr, 101 op/s 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: 145.6 scrub starts 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: 145.6 scrub ok 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: 145.1 scrub starts 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: 145.1 scrub ok 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1489401826' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi114-82405-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:16 smithi114 ceph-mon[42466]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:43:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:17 smithi119 ceph-mon[33714]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:43:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/207393099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:17.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:17 smithi119 ceph-mon[33714]: from='client.55541 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[30312]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/207393099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[30312]: from='client.55541 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[42466]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/207393099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:17 smithi114 ceph-mon[42466]: from='client.55541 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:18 smithi119 ceph-mon[33714]: pgmap v164: 492 pgs: 32 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-24T11:43:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:18 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:18 smithi119 ceph-mon[33714]: from='client.55541 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:18 smithi119 ceph-mon[33714]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:43:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3471552687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:18.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:18 smithi119 ceph-mon[33714]: from='client.55544 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[30312]: pgmap v164: 492 pgs: 32 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-24T11:43:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[30312]: from='client.55541 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[30312]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:43:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3471552687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[30312]: from='client.55544 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[42466]: pgmap v164: 492 pgs: 32 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-24T11:43:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[42466]: from='client.55541 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi114-82433-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[42466]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:43:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3471552687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:18 smithi114 ceph-mon[42466]: from='client.55544 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:19.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:19 smithi119 ceph-mon[33714]: from='client.55544 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:19.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:19 smithi119 ceph-mon[33714]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:43:19.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:19.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:19 smithi114 ceph-mon[42466]: from='client.55544 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:19.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:19 smithi114 ceph-mon[42466]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:43:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:19 smithi114 ceph-mon[30312]: from='client.55544 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi114-82405-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:19 smithi114 ceph-mon[30312]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:43:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: pgmap v167: 492 pgs: 32 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-24T11:43:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:43:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1918445872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:20 smithi119 ceph-mon[33714]: from='client.48589 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: pgmap v167: 492 pgs: 32 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1918445872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[30312]: from='client.48589 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: pgmap v167: 492 pgs: 32 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1918445872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:20.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:20 smithi114 ceph-mon[42466]: from='client.48589 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.48589 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:43:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3784900375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.55556 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4056472536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82405-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:43:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.48589 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:43:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3784900375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.55556 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4056472536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82405-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.48589 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi114-82433-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1768288333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:21.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:43:21.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3784900375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]: dispatch 2024-03-24T11:43:21.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.55556 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4056472536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82405-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:43:21.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:21 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (4654 ms) 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (5452 ms) 2024-03-24T11:43:21.746 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2350 ms) 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2035 ms) 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (14491 ms total) 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (6109 ms) 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4414 ms) 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5753 ms) 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (16276 ms total) 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-24T11:43:21.747 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3004 ms) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2010 ms) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1966 ms) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1998 ms) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8978 ms total) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4020 ms) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4026 ms) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8046 ms total) 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: 2024-03-24T11:43:21.748 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-24T11:43:21.749 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (73810 ms total) 2024-03-24T11:43:21.749 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-24T11:43:22.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:43:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:43:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: pgmap v170: 516 pgs: 96 unknown, 420 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.55556 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4056472536' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82405-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:43:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2"}]: dispatch 2024-03-24T11:43:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:22.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:22 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: pgmap v170: 516 pgs: 96 unknown, 420 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.55556 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4056472536' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82405-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2"}]: dispatch 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:22.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: pgmap v170: 516 pgs: 96 unknown, 420 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.48550 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi114-82736-15"}]': finished 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.55556 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82839-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4056472536' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82405-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2"}]: dispatch 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:22.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:22 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4204798555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.48610 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:23 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:43:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4204798555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.48610 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi114-82657-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4204798555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.48610 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:23 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-24T11:43:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: pgmap v173: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: from='client.48610 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-24T11:43:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:43:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2080843887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: from='client.48625 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:43:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:24 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:43:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: pgmap v173: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: from='client.48610 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-24T11:43:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:43:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2080843887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: from='client.48625 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: pgmap v173: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: from='client.48610 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi114-82433-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2080843887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: from='client.48625 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:43:24.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-24T11:43:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:43:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:43:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.48625 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:25 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.48625 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-24T11:43:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:43:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:43:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:43:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.48625 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi114-82405-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4088235741' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:25.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:25 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: pgmap v176: 452 pgs: 32 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:43:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]': finished 2024-03-24T11:43:26.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:26.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:43:26.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3540813606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:26.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:26.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: from='client.48640 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:26.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:26 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: pgmap v176: 452 pgs: 32 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]': finished 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3540813606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: from='client.48640 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: pgmap v176: 452 pgs: 32 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:26.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: from='client.24724 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi114-82657-1","app":"app1","key":"key1"}]': finished 2024-03-24T11:43:26.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:26.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:43:26.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3540813606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:26.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:26.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: from='client.48640 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:26.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:26 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:27.016 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-24T11:43:27.016 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-24T11:43:27.016 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-24T11:43:27.016 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-24T11:43:27.016 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-24T11:43:27.016 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-24T11:43:27.016 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: seed 82657 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (320 ms) 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (16 ms) 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (11 ms) 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (10 ms) 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-24T11:43:27.017 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (4 ms) 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (6 ms) 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (4 ms) 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12 ms) 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (11 ms) 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-24T11:43:27.018 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1531 ms) 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (415 ms) 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: done waiting, doing copies 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: done waiting 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60802 ms) 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (12 ms) 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (2 ms) 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4990 ms) 2024-03-24T11:43:27.019 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-24T11:43:27.020 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-24T11:43:27.020 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-24T11:43:27.020 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-24T11:43:27.020 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-24T11:43:27.020 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-24T11:43:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: from='client.48640 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:43:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2071784272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: from='client.48655 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82657-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: from='client.48640 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2071784272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: from='client.48655 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82657-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: from='client.48640 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2071784272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: from='client.48655 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:27.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82657-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:28 smithi119 ceph-mon[33714]: pgmap v179: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:28 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:28 smithi119 ceph-mon[33714]: from='client.48655 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82657-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:28 smithi119 ceph-mon[33714]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:43:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82657-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[30312]: pgmap v179: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[30312]: from='client.48655 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82657-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[30312]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82657-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[42466]: pgmap v179: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi114-82839-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[42466]: from='client.48655 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi114-82405-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82657-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[42466]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:43:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82657-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:29 smithi119 ceph-mon[33714]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:43:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4239787384' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:29 smithi119 ceph-mon[33714]: from='client.48661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:29 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[30312]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:43:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4239787384' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[30312]: from='client.48661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[42466]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:43:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4239787384' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[42466]: from='client.48661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:29 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: pgmap v182: 428 pgs: 40 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82657-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='client.48661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1957948275' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='client.55622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: pgmap v182: 428 pgs: 40 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82657-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='client.48661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1957948275' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='client.55622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:43:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: pgmap v182: 428 pgs: 40 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82657-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='client.48661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi114-82433-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:43:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1957948275' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='client.55622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: pgmap v184: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: pgmap v185: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: from='client.55622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: Health check update: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:31.613 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: pgmap v184: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: pgmap v185: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: from='client.55622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: Health check update: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: pgmap v184: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: pgmap v185: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-24T11:43:31.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:31.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: from='client.55622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi114-82405-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:31.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:43:31.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: Health check update: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-24T11:43:31.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:31.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:31.911 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:43:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:43:32.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:32.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:32.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[42466]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:43:32.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1994810448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi114-82433-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:32.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:32.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:32.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:32.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[30312]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:43:32.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1994810448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi114-82433-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:32.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:32.776 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 22 tests from LibRadosMisc api_snapshots_pp: Running main() from gmock_main.cc 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: seed 82839 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (4657 ms) 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (5451 ms) 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2351 ms) 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2022 ms) 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-24T11:43:32.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (5175 ms) 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (19656 ms total) 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4261 ms) 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4750 ms) 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6256 ms) 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4011 ms) 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2052 ms) 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-24T11:43:32.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1667-gdb0330b1/rpm/el9/BUILD/ceph-19.0.0-1667-gdb0330b1/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi114-82839-7 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17915 ms) 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (39245 ms total) 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5070 ms) 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5079 ms total) 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: 2024-03-24T11:43:32.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-24T11:43:32.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-24T11:43:32.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3006 ms) 2024-03-24T11:43:32.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-24T11:43:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:32 smithi119 ceph-mon[33714]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:43:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1994810448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi114-82433-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: pgmap v188: 468 pgs: 32 creating+peering, 32 unknown, 404 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 631 B/s wr, 2 op/s 2024-03-24T11:43:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1994810448' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi114-82433-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1285559903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi114-82405-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: pgmap v188: 468 pgs: 32 creating+peering, 32 unknown, 404 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 631 B/s wr, 2 op/s 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1994810448' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi114-82433-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:43:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1285559903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi114-82405-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: pgmap v188: 468 pgs: 32 creating+peering, 32 unknown, 404 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 631 B/s wr, 2 op/s 2024-03-24T11:43:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1994810448' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi114-82433-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82657-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:43:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1285559903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi114-82405-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1285559903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi114-82405-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:34.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:34 smithi114 ceph-mon[30312]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:43:34.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:34.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1285559903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi114-82405-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:34.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:34 smithi114 ceph-mon[42466]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:43:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1285559903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi114-82405-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:34 smithi119 ceph-mon[33714]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:43:34.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:43:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:43:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[30312]: pgmap v191: 436 pgs: 32 unknown, 404 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[30312]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/556409289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi114-82433-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[42466]: pgmap v191: 436 pgs: 32 unknown, 404 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:35.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[42466]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:43:35.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/556409289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi114-82433-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:35.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:35 smithi119 ceph-mon[33714]: pgmap v191: 436 pgs: 32 unknown, 404 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:43:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:35 smithi119 ceph-mon[33714]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:43:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/556409289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi114-82433-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/556409289' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi114-82433-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[30312]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:43:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4141095177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi114-82405-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/556409289' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi114-82433-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[42466]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:43:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4141095177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi114-82405-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:36.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/556409289' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi114-82433-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:36 smithi119 ceph-mon[33714]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:43:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]: dispatch 2024-03-24T11:43:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4141095177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi114-82405-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[30312]: pgmap v194: 460 pgs: 64 unknown, 396 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4141095177' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi114-82405-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[30312]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:43:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[42466]: pgmap v194: 460 pgs: 64 unknown, 396 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:37.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:37.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:37.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:37.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4141095177' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi114-82405-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:37.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:37 smithi114 ceph-mon[42466]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:43:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:37 smithi119 ceph-mon[33714]: pgmap v194: 460 pgs: 64 unknown, 396 active+clean; 592 KiB data, 437 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:37 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3938620764' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82657-24"}]': finished 2024-03-24T11:43:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4141095177' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi114-82405-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:37 smithi119 ceph-mon[33714]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:43:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:43:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3751466897' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2389830033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: from='client.48694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3751466897' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2389830033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: from='client.48694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:38 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:43:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3751466897' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2389830033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: from='client.48694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:38.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:38 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:39.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: pgmap v197: 452 pgs: 64 unknown, 388 active+clean; 592 KiB data, 481 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:39.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:39.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.48694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/383780009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[30312]: from='client.55658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: pgmap v197: 452 pgs: 64 unknown, 388 active+clean; 592 KiB data, 481 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:39.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.48694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:39.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:39.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:39.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:43:39.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:39.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:39.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/383780009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:39.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:39 smithi114 ceph-mon[42466]: from='client.55658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: pgmap v197: 452 pgs: 64 unknown, 388 active+clean; 592 KiB data, 481 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.48694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi114-82433-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi114-82657-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:43:39.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1920724903' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:39.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]: dispatch 2024-03-24T11:43:39.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/383780009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:39.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:39 smithi119 ceph-mon[33714]: from='client.55658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.55658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:43:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.55658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:40.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:40 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.55601 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi114-82839-16"}]': finished 2024-03-24T11:43:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.55658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi114-82405-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:43:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:40 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: pgmap v200: 420 pgs: 32 unknown, 388 active+clean; 592 KiB data, 481 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3883258975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/512083194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82433-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:41.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:41.623 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi114-82405-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: pgmap v200: 420 pgs: 32 unknown, 388 active+clean; 592 KiB data, 481 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:43:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3883258975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/512083194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82433-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi114-82405-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: pgmap v200: 420 pgs: 32 unknown, 388 active+clean; 592 KiB data, 481 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:41.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3883258975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/512083194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82433-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:41.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi114-82405-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:42.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:43:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:43:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/512083194' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82433-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[30312]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi114-82405-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[30312]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:43:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi114-82405-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:42.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:42.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/512083194' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82433-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:42.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[42466]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:42.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi114-82405-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:42.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[42466]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:43:42.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi114-82405-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/512083194' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi114-82433-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:42 smithi119 ceph-mon[33714]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi114-82657-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi114-82405-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:42 smithi119 ceph-mon[33714]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:43:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi114-82405-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[30312]: pgmap v203: 452 pgs: 64 creating+peering, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[30312]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:43:43.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:43.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[42466]: pgmap v203: 452 pgs: 64 creating+peering, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:43.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:43.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:43.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:43.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:43 smithi114 ceph-mon[42466]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:43:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:43.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:43 smithi119 ceph-mon[33714]: pgmap v203: 452 pgs: 64 creating+peering, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:43.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:43.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:43 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:43.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:43 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:43.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:43 smithi119 ceph-mon[33714]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:43:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi114-82405-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[30312]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:43:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2793742606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2978002148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[30312]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[30312]: from='client.48733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:43:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi114-82405-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[42466]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2793742606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2978002148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[42466]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:44 smithi114 ceph-mon[42466]: from='client.48733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi114-82405-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:44 smithi119 ceph-mon[33714]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:43:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2793742606' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2978002148' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:44 smithi119 ceph-mon[33714]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:44.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:44 smithi119 ceph-mon[33714]: from='client.48733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[30312]: pgmap v206: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[30312]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[30312]: from='client.48733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[30312]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:43:45.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[42466]: pgmap v206: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:45.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:45.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[42466]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:45.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[42466]: from='client.48733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:45.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:45 smithi114 ceph-mon[42466]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:43:45.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:45 smithi119 ceph-mon[33714]: pgmap v206: 468 pgs: 80 unknown, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:45.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:45.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:45 smithi119 ceph-mon[33714]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi114-82433-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:45.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:45 smithi119 ceph-mon[33714]: from='client.48733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi114-82657-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:45.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:45 smithi119 ceph-mon[33714]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:43:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:43:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:46 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:43:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:46.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:46.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:46 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: pgmap v209: 396 pgs: 8 unknown, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:43:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1353530678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi114-82433-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: pgmap v209: 396 pgs: 8 unknown, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:47.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1353530678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi114-82433-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:47.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:47.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:47.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:47.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: pgmap v209: 396 pgs: 8 unknown, 388 active+clean; 592 KiB data, 486 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:47.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:47.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:47.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:47.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:43:47.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]: dispatch 2024-03-24T11:43:47.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:47.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1353530678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi114-82433-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:47.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:47.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:47.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosS api_aio: Running main() from gmock_main.cc 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [----------] Global test environment set-up. 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10397 ms) 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (4045 ms) 2024-03-24T11:43:47.998 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (5725 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3336 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2861 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4072 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (4171 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3442 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2458 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3153 ms) 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-24T11:43:47.999 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3470 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3760 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3027 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2947 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.Flush (2995 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3015 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2999 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3007 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3013 ms) 2024-03-24T11:43:48.000 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3018 ms) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3016 ms) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3251 ms) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2770 ms) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3017 ms) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2897 ms) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3022 ms) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [----------] 26 tests from LibRadosAio (92885 ms total) 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-24T11:43:48.001 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1353530678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi114-82433-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1353530678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi114-82433-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:48.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:48 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:48.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:48.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:48.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/932434361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi114-82405-27"}]': finished 2024-03-24T11:43:48.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1353530678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi114-82433-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:48.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:48 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: pgmap v212: 428 pgs: 32 unknown, 396 active+clean; 603 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]': finished 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:43:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: pgmap v212: 428 pgs: 32 unknown, 396 active+clean; 603 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]': finished 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:43:49.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:49 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: pgmap v212: 428 pgs: 32 unknown, 396 active+clean; 603 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T11:43:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:49.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:49.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi114-82657-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:49.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-13"}]': finished 2024-03-24T11:43:49.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi114-82405-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:49.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:49.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:43:49.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:49 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[30312]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:43:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1748457727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[30312]: from='client.55712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[42466]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:43:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1748457727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:50 smithi114 ceph-mon[42466]: from='client.55712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:50 smithi119 ceph-mon[33714]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:43:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1748457727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:50 smithi119 ceph-mon[33714]: from='client.55712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:51.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: pgmap v215: 404 pgs: 40 unknown, 364 active+clean; 602 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-24T11:43:51.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:51.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:51.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: from='client.55712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:51.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:43:51.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: pgmap v215: 404 pgs: 40 unknown, 364 active+clean; 602 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: from='client.55712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:51.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:51 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:51.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:43:51.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: pgmap v215: 404 pgs: 40 unknown, 364 active+clean; 602 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-24T11:43:51.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:51.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi114-82405-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:51.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: from='client.55712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi114-82433-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:51.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:43:51.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:51.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:51.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:51.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:51 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:52.706 INFO:tasks.workunit.client.0.smithi114.stdout:PP (68158 ms total) 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (100 ms) 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (100 ms total) 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (44 ms) 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (48 ms total) 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-24T11:43:52.707 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (34 ms) 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (6 ms) 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (40 ms total) 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (45 ms) 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (9 ms) 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (54 ms total) 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1035 ms) 2024-03-24T11:43:52.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1035 ms total) 2024-03-24T11:43:52.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: 2024-03-24T11:43:52.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-24T11:43:52.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104761 ms total) 2024-03-24T11:43:52.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-24T11:43:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[30312]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:43:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:52.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[42466]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:43:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:52 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:52 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:52.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1186501321' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:52.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:52 smithi119 ceph-mon[33714]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:43:52.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:52 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]: dispatch 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: pgmap v218: 404 pgs: 40 creating+peering, 364 active+clean; 609 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2507302523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.55715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: pgmap v218: 404 pgs: 40 creating+peering, 364 active+clean; 609 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2507302523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:53.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.55715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:53.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:53.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:53 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: pgmap v218: 404 pgs: 40 creating+peering, 364 active+clean; 609 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.48742 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi114-82657-36"}]': finished 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2507302523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.55715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:53.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:53 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:43:54.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:43:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:54.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:54.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:54.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.55715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:54.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:43:54.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.55715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:54.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:54.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:54 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.55715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi114-82433-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:54.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:43:54.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:43:54.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:54.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1449890225' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:54.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:54.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:54 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]: dispatch 2024-03-24T11:43:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: pgmap v221: 428 pgs: 32 unknown, 32 creating+peering, 364 active+clean; 609 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2024-03-24T11:43:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]': finished 2024-03-24T11:43:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]: dispatch 2024-03-24T11:43:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: pgmap v221: 428 pgs: 32 unknown, 32 creating+peering, 364 active+clean; 609 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]': finished 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:55.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]: dispatch 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]: dispatch 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:55.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:55 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:55.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: pgmap v221: 428 pgs: 32 unknown, 32 creating+peering, 364 active+clean; 609 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 3 op/s 2024-03-24T11:43:55.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:55.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-15"}]': finished 2024-03-24T11:43:55.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.55706 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi114-82405-28"}]': finished 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]: dispatch 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]: dispatch 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:55.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:55 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:43:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]': finished 2024-03-24T11:43:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:43:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2148971685' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi114-82433-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]': finished 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:56.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:56.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2148971685' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi114-82433-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:56.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:56 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:56.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:56.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:43:56.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-15", "mode": "writeback"}]': finished 2024-03-24T11:43:56.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi114-82405-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:43:56.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:43:56.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:56.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:56.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:43:56.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2148971685' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi114-82433-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:56.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:56 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout:napshotsECPP.SnapRemovePP (2006 ms) 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2019 ms) 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2008 ms) 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9039 ms total) 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-24T11:43:57.708 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4045 ms) 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3997 ms) 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3945 ms) 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11987 ms total) 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (109737 ms total) 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-24T11:43:57.709 INFO:tasks.workunit.client.0.smithi114.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: pgmap v224: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2148971685' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi114-82433-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: pgmap v224: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2148971685' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi114-82433-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:57.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:57.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: pgmap v224: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:57.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:57.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2148971685' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi114-82433-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:43:57.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:57.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3514555568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:57.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:43:57.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]: dispatch 2024-03-24T11:43:57.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:57.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:43:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi114-82405-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:43:58.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: from='client.55673 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi114-82839-21"}]': finished 2024-03-24T11:43:58.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:43:58.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:43:58.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:58.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]: dispatch 2024-03-24T11:43:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[30312]: pgmap v227: 396 pgs: 8 unknown, 388 active+clean; 585 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]': finished 2024-03-24T11:43:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[30312]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:43:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1596667590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[30312]: from='client.48781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[42466]: pgmap v227: 396 pgs: 8 unknown, 388 active+clean; 585 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]': finished 2024-03-24T11:43:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[42466]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:43:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1596667590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:59.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:43:59 smithi114 ceph-mon[42466]: from='client.48781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:59 smithi119 ceph-mon[33714]: pgmap v227: 396 pgs: 8 unknown, 388 active+clean; 585 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:43:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:43:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:59 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:43:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-15"}]': finished 2024-03-24T11:43:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:59 smithi119 ceph-mon[33714]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:43:59.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1596667590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:43:59.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:43:59 smithi119 ceph-mon[33714]: from='client.48781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[30312]: from='client.48781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[30312]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:44:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:00.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:00.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[42466]: from='client.48781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[42466]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:44:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:00 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:00.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:00.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:00 smithi119 ceph-mon[33714]: from='client.48781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi114-82433-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:00.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:00 smithi119 ceph-mon[33714]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:44:00.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:00.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:00 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: pgmap v230: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:01.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: pgmap v230: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:44:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:44:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: pgmap v230: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4006395383' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]: dispatch 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:01.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:44:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:44:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:44:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:44:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2946777696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.55739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi114-82405-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2946777696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.55739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:02.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:02.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:02.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi114-82405-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.48772 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi114-82405-29"}]': finished 2024-03-24T11:44:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2946777696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.55739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi114-82405-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: pgmap v233: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.55739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi114-82405-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi114-82405-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:03.974 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (10093 ms) 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (26706 ms) 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6880 ms) 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3443 ms) 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2773 ms) 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-24T11:44:03.975 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2998 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2981 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2992 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3237 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2752 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3012 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3019 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3023 ms) 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-24T11:44:03.976 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3007 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3013 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3003 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3013 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3018 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2907 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3259 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2772 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3015 ms) 2024-03-24T11:44:03.977 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3011 ms) 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2945 ms) 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2993 ms) 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3018 ms) 2024-03-24T11:44:03.978 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: pgmap v233: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.55739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi114-82405-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:03.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi114-82405-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: pgmap v233: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.55739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi114-82433-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:03.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi114-82405-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:03.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:44:03.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:03.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi114-82405-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:03.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:03.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:03.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]': finished 2024-03-24T11:44:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]: dispatch 2024-03-24T11:44:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:04 smithi119 ceph-mon[33714]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:44:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]: dispatch 2024-03-24T11:44:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:04 smithi119 ceph-mon[33714]: pgmap v236: 388 pgs: 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:04.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:44:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:44:04.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:04.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]': finished 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]: dispatch 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[30312]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]: dispatch 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[30312]: pgmap v236: 388 pgs: 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-17"}]': finished 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]: dispatch 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[42466]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]: dispatch 2024-03-24T11:44:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:04 smithi114 ceph-mon[42466]: pgmap v236: 388 pgs: 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi114-82405-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:05.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]': finished 2024-03-24T11:44:05.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:44:05.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2837979066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:05.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: from='client.55751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:05.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:05.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:05.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:05.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:05.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi114-82405-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]': finished 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2837979066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: from='client.55751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi114-82405-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:05.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-17", "mode": "writeback"}]': finished 2024-03-24T11:44:05.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:44:05.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2837979066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:05.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: from='client.55751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:05.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:05.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:06 smithi119 ceph-mon[33714]: from='client.55751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:06.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:06.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:06 smithi119 ceph-mon[33714]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:44:06.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:06.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:06.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:06 smithi119 ceph-mon[33714]: pgmap v239: 428 pgs: 40 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[30312]: from='client.55751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[30312]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:44:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:06.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[30312]: pgmap v239: 428 pgs: 40 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:06.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:06.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[42466]: from='client.55751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi114-82433-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:06.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:06.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[42466]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:44:06.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:06.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]: dispatch 2024-03-24T11:44:06.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:06 smithi114 ceph-mon[42466]: pgmap v239: 428 pgs: 40 unknown, 32 creating+peering, 356 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:07 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:07 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]': finished 2024-03-24T11:44:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:07 smithi119 ceph-mon[33714]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:44:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]': finished 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[30312]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:07.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-17"}]': finished 2024-03-24T11:44:07.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[42466]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:44:07.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:08 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:08 smithi119 ceph-mon[33714]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:44:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/764161296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi114-82433-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:08 smithi119 ceph-mon[33714]: pgmap v242: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:08.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:08.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:08.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:08.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[30312]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:44:08.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:08.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/764161296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi114-82433-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:08.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[30312]: pgmap v242: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:08.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:08.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:08.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:08.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[42466]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:44:08.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]: dispatch 2024-03-24T11:44:08.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/764161296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi114-82433-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:08.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:08 smithi114 ceph-mon[42466]: pgmap v242: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/764161296' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi114-82433-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:09.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:09 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/764161296' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi114-82433-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2779834166' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi114-82405-30"}]': finished 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/764161296' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi114-82433-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:09.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:09.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:09 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:10.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:10.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:10.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:10.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:44:10.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:10.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:10.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:10.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:10.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:10 smithi119 ceph-mon[33714]: pgmap v245: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:10.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[30312]: pgmap v245: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi114-82405-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:44:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:10.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:10.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:10 smithi114 ceph-mon[42466]: pgmap v245: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:44:11.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:44:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:11.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-mon[33714]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:44:11.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:11.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2815761679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:11.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:11 smithi119 ceph-mon[33714]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[30312]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:44:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2815761679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[30312]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[42466]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2815761679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:11 smithi114 ceph-mon[42466]: from='client.48811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:12.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:12.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:12.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]': finished 2024-03-24T11:44:12.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:12.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:44:12.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]: dispatch 2024-03-24T11:44:12.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]: dispatch 2024-03-24T11:44:12.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:12 smithi119 ceph-mon[33714]: pgmap v248: 428 pgs: 8 unknown, 32 creating+peering, 388 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]': finished 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]: dispatch 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]: dispatch 2024-03-24T11:44:12.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[42466]: pgmap v248: 428 pgs: 8 unknown, 32 creating+peering, 388 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi114-82405-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-19"}]': finished 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: from='client.48811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi114-82433-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]: dispatch 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]: dispatch 2024-03-24T11:44:12.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:12 smithi114 ceph-mon[30312]: pgmap v248: 428 pgs: 8 unknown, 32 creating+peering, 388 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:13 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:13 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:13 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]': finished 2024-03-24T11:44:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:13 smithi119 ceph-mon[33714]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:44:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:13 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]': finished 2024-03-24T11:44:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[42466]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:44:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:13.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:13.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:13.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:13.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-19", "mode": "writeback"}]': finished 2024-03-24T11:44:13.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[30312]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:44:13.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:13.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:13 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:44:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:14.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:14.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/540781006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:14.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: from='client.55775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:14.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:14 smithi119 ceph-mon[33714]: pgmap v251: 420 pgs: 32 unknown, 388 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:44:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:44:14.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:14.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:14.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:14.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/540781006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: from='client.55775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[30312]: pgmap v251: 420 pgs: 32 unknown, 388 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:14.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:14.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:44:14.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:14.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]: dispatch 2024-03-24T11:44:14.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:14.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/540781006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:14.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: from='client.55775 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:14.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:14 smithi114 ceph-mon[42466]: pgmap v251: 420 pgs: 32 unknown, 388 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]': finished 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.55775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/540781006' entity='client.admin' cmd=[{ 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "prefix": "osd pool set", 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "var": "eio", 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "val": "true" 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: }]: dispatch 2024-03-24T11:44:15.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: from='client.55775 ' entity='client.admin' cmd=[{ 2024-03-24T11:44:15.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "prefix": "osd pool set", 2024-03-24T11:44:15.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:15.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "var": "eio", 2024-03-24T11:44:15.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: "val": "true" 2024-03-24T11:44:15.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:15 smithi119 ceph-mon[33714]: }]: dispatch 2024-03-24T11:44:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]': finished 2024-03-24T11:44:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.55775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/540781006' entity='client.admin' cmd=[{ 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "prefix": "osd pool set", 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "var": "eio", 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "val": "true" 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: }]: dispatch 2024-03-24T11:44:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: from='client.55775 ' entity='client.admin' cmd=[{ 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "prefix": "osd pool set", 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "var": "eio", 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: "val": "true" 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[42466]: }]: dispatch 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-19"}]': finished 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.55775 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi114-82433-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1789716970' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:44:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]: dispatch 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/540781006' entity='client.admin' cmd=[{ 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "prefix": "osd pool set", 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "var": "eio", 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "val": "true" 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: }]: dispatch 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: from='client.55775 ' entity='client.admin' cmd=[{ 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "prefix": "osd pool set", 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "var": "eio", 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: "val": "true" 2024-03-24T11:44:15.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:15 smithi114 ceph-mon[30312]: }]: dispatch 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.55775 ' entity='client.admin' cmd='[{ 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: "prefix": "osd pool set", 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: "var": "eio", 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: "val": "true" 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: }]': finished 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:16.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:16.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:16 smithi119 ceph-mon[33714]: pgmap v254: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.55775 ' entity='client.admin' cmd='[{ 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: "prefix": "osd pool set", 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: "var": "eio", 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: "val": "true" 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: }]': finished 2024-03-24T11:44:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[30312]: pgmap v254: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.55763 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi114-82405-31"}]': finished 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.55775 ' entity='client.admin' cmd='[{ 2024-03-24T11:44:16.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: "prefix": "osd pool set", 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: "pool": "PoolEIOFlag_smithi114-82433-33", 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: "var": "eio", 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: "val": "true" 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: }]': finished 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:16.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:16 smithi114 ceph-mon[42466]: pgmap v254: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:17 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:17 smithi119 ceph-mon[33714]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T11:44:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:17.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:17.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:17 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:17.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[30312]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T11:44:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi114-82405-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[42466]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T11:44:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T11:44:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3943825726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi114-82433-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:18 smithi119 ceph-mon[33714]: pgmap v257: 420 pgs: 64 unknown, 356 active+clean; 585 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3943825726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi114-82433-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[30312]: pgmap v257: 420 pgs: 64 unknown, 356 active+clean; 585 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3943825726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi114-82433-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:18 smithi114 ceph-mon[42466]: pgmap v257: 420 pgs: 64 unknown, 356 active+clean; 585 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:44:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3943825726' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi114-82433-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:19.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T11:44:19.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:19.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:19 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3943825726' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi114-82433-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi114-82405-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3943825726' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi114-82433-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:19.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:19 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:20 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]': finished 2024-03-24T11:44:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]: dispatch 2024-03-24T11:44:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:20 smithi119 ceph-mon[33714]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T11:44:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:20 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]: dispatch 2024-03-24T11:44:20.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:20 smithi119 ceph-mon[33714]: pgmap v260: 396 pgs: 40 unknown, 356 active+clean; 585 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:44:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]': finished 2024-03-24T11:44:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]: dispatch 2024-03-24T11:44:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[30312]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T11:44:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]: dispatch 2024-03-24T11:44:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[30312]: pgmap v260: 396 pgs: 40 unknown, 356 active+clean; 585 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:44:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-21"}]': finished 2024-03-24T11:44:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]: dispatch 2024-03-24T11:44:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[42466]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T11:44:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]: dispatch 2024-03-24T11:44:20.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:20 smithi114 ceph-mon[42466]: pgmap v260: 396 pgs: 40 unknown, 356 active+clean; 585 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:44:21.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:44:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]': finished 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1504016810' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.55793 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]': finished 2024-03-24T11:44:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T11:44:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1504016810' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.55793 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-21", "mode": "writeback"}]': finished 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1504016810' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:21.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.55793 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:21.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:21.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.55793 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T11:44:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:22.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:22.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:22.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:22.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:22 smithi119 ceph-mon[33714]: pgmap v263: 420 pgs: 32 creating+peering, 388 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T11:44:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.55793 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[42466]: pgmap v263: 420 pgs: 32 creating+peering, 388 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:22.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.55793 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi114-82433-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:22.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:22.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T11:44:22.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:22.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]: dispatch 2024-03-24T11:44:22.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1998244122' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:22.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]: dispatch 2024-03-24T11:44:22.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:22 smithi114 ceph-mon[30312]: pgmap v263: 420 pgs: 32 creating+peering, 388 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T11:44:23.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:23.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:23.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]': finished 2024-03-24T11:44:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T11:44:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:23.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi114-82405-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]': finished 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi114-82405-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-21"}]': finished 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: from='client.55781 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi114-82405-32"}]': finished 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi114-82405-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi114-82405-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:24 smithi119 ceph-mon[33714]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T11:44:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1881069947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi114-82405-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:24 smithi119 ceph-mon[33714]: from='client.48844 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:24.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:24 smithi119 ceph-mon[33714]: pgmap v266: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:44:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:44:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi114-82405-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[30312]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T11:44:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1881069947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi114-82405-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[30312]: from='client.48844 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[30312]: pgmap v266: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi114-82405-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[42466]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1881069947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi114-82405-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[42466]: from='client.48844 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:24 smithi114 ceph-mon[42466]: pgmap v266: 388 pgs: 32 unknown, 356 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:25 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:25 smithi119 ceph-mon[33714]: from='client.48844 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:25 smithi119 ceph-mon[33714]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T11:44:25.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:25.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[30312]: from='client.48844 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[30312]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T11:44:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:25.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:25.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:25.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[42466]: from='client.48844 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:25.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[42466]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T11:44:25.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:25.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi114-82405-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:26 smithi119 ceph-mon[33714]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T11:44:26.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1519535634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:26.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:26 smithi119 ceph-mon[33714]: from='client.55811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:26.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:26 smithi119 ceph-mon[33714]: pgmap v269: 460 pgs: 104 unknown, 356 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi114-82405-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[42466]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T11:44:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1519535634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[42466]: from='client.55811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[42466]: pgmap v269: 460 pgs: 104 unknown, 356 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:26.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:26.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi114-82405-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:26.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:26.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[30312]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T11:44:26.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1519535634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:26.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[30312]: from='client.55811 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:26.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:26 smithi114 ceph-mon[30312]: pgmap v269: 460 pgs: 104 unknown, 356 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:27 smithi119 ceph-mon[33714]: from='client.55811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:27 smithi119 ceph-mon[33714]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T11:44:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:27.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[30312]: from='client.55811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[30312]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T11:44:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:27.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:27.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[42466]: from='client.55811 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:27.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[42466]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T11:44:27.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:27.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T11:44:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/773741239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:28.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:28 smithi119 ceph-mon[33714]: pgmap v272: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/773741239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[30312]: pgmap v272: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/773741239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:28 smithi114 ceph-mon[42466]: pgmap v272: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:44:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/773741239' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]': finished 2024-03-24T11:44:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]: dispatch 2024-03-24T11:44:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T11:44:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]: dispatch 2024-03-24T11:44:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:29.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:29.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:29.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/773741239' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:29.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]': finished 2024-03-24T11:44:29.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]: dispatch 2024-03-24T11:44:29.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]: dispatch 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/773741239' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi114-82433-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-23"}]': finished 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]: dispatch 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]: dispatch 2024-03-24T11:44:29.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]: dispatch 2024-03-24T11:44:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]': finished 2024-03-24T11:44:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: pgmap v275: 452 pgs: 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:44:30.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:30 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]': finished 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: pgmap v275: 452 pgs: 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-23", "mode": "writeback"}]': finished 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2965957847' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi114-82405-33"}]': finished 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T11:44:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: pgmap v275: 452 pgs: 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:44:30.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:30 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:31.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:44:31.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: pgmap v276: 452 pgs: 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:31.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:31 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:31.890 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:31.890 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: pgmap v276: 452 pgs: 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:31.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: pgmap v276: 452 pgs: 32 creating+peering, 420 active+clean; 586 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi114-82405-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:31.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:31 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:32 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:32 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:32 smithi119 ceph-mon[33714]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T11:44:32.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:32 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[42466]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:32.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:32.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:32.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:32.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[30312]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T11:44:32.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:32 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]: dispatch 2024-03-24T11:44:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: pgmap v279: 388 pgs: 388 active+clean; 585 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 518 B/s wr, 5 op/s 2024-03-24T11:44:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]': finished 2024-03-24T11:44:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T11:44:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3612089217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: from='client.55829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: pgmap v279: 388 pgs: 388 active+clean; 585 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 518 B/s wr, 5 op/s 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]': finished 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3612089217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: from='client.55829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: pgmap v279: 388 pgs: 388 active+clean; 585 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 518 B/s wr, 5 op/s 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi114-82405-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-23"}]': finished 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3612089217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: from='client.55829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:44:33.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.55829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]': finished 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:34 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:34.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:44:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:44:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.55829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]': finished 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.55829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]': finished 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T11:44:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:34.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:34.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:34 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: pgmap v282: 396 pgs: 40 unknown, 356 active+clean; 585 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]': finished 2024-03-24T11:44:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T11:44:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2666984453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.55835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: pgmap v282: 396 pgs: 40 unknown, 356 active+clean; 585 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]': finished 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2666984453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:36.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.55835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:36.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:36.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:35 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: pgmap v282: 396 pgs: 40 unknown, 356 active+clean; 585 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]': finished 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2666984453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.55835 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:36.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:35 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]: dispatch 2024-03-24T11:44:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='client.55835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]': finished 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi114-82498-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='client.55835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]': finished 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi114-82498-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-24T11:44:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='client.55835 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pg_num","val":"11"}]': finished 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3313390805' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]: dispatch 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:37.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi114-82498-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: pgmap v285: 452 pgs: 452 active+clean; 586 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s wr, 7 op/s 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi114-82498-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/261283395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.55841 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi114-82405-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]': finished 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.55841 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi114-82405-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi114-82405-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: pgmap v285: 452 pgs: 452 active+clean; 586 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s wr, 7 op/s 2024-03-24T11:44:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi114-82498-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-24T11:44:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/261283395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.55841 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi114-82405-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]': finished 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.55841 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:38.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: pgmap v285: 452 pgs: 452 active+clean; 586 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s wr, 7 op/s 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.55823 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi114-82405-34"}]': finished 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi114-82498-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/261283395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.55841 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi114-82405-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.24697 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi114-82498-1","var":"pgp_num","val":"11"}]': finished 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.55841 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:38.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi114-82405-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi114-82405-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi114-82405-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi114-82405-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/99746608' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-24T11:44:38.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]': finished 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]: dispatch 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: pgmap v289: 484 pgs: 64 unknown, 420 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/817437862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]: dispatch 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]': finished 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: pgmap v289: 484 pgs: 64 unknown, 420 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/817437862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:39.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:39.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:39 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-25"}]': finished 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: pgmap v289: 484 pgs: 64 unknown, 420 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/817437862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:40.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:39 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi114-82405-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]': finished 2024-03-24T11:44:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi114-82405-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]': finished 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T11:44:40.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:40.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:40.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:41.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:41.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi114-82405-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:41.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-25", "mode": "writeback"}]': finished 2024-03-24T11:44:41.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:41.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi114-82498-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:41.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T11:44:41.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:41.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:41.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:41.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:41.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[42466]: pgmap v292: 524 pgs: 104 unknown, 420 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:41.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[42466]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T11:44:41.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2287519486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:41.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:41.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[42466]: from='client.55865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:41.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:41.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:41.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[30312]: pgmap v292: 524 pgs: 104 unknown, 420 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:41.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[30312]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T11:44:41.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2287519486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:41.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:41.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:41 smithi114 ceph-mon[30312]: from='client.55865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-mon[33714]: pgmap v292: 524 pgs: 104 unknown, 420 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-mon[33714]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T11:44:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2287519486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]: dispatch 2024-03-24T11:44:42.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-mon[33714]: from='client.55865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:42.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:44:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:44:42.953 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAiexpected=8 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:b29083e3:::5:head 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:b29083e3:::5:head expected=14:b29083e3:::5:head 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:b29083e3:::5:head -> 5 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=5 expected=5 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:89d3ae78:::11:head 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:89d3ae78:::11:head expected=14:89d3ae78:::11:head 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:89d3ae78:::11:head -> 11 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=11 expected=11 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:6cac518f:::0:head 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:6cac518f:::0:head expected=14:6cac518f:::0:head 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:6cac518f:::0:head -> 0 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=0 expected=0 2024-03-24T11:44:42.954 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:5c6b0b28:::7:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:5c6b0b28:::7:head expected=14:5c6b0b28:::7:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:5c6b0b28:::7:head -> 7 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=7 expected=7 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:566253c9:::13:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:566253c9:::13:head expected=14:566253c9:::13:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:566253c9:::13:head -> 13 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=13 expected=13 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:52ea6a34:::10:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:52ea6a34:::10:head expected=14:52ea6a34:::10:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:52ea6a34:::10:head -> 10 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=10 expected=10 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:62a1935d:::14:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:62a1935d:::14:head expected=14:62a1935d:::14:head 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:62a1935d:::14:head -> 14 2024-03-24T11:44:42.955 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=14 expected=14 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : seek to 14:02547ec2:::1:head 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : cursor()=14:02547ec2:::1:head expected=14:02547ec2:::1:head 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: > 14:02547ec2:::1:head -> 1 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: : entry=1 expected=1 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (235 ms) 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (136487 ms) 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: split 0/5 -> MIN 14:33333333::::head 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: split 1/5 -> 14:33333333::::head 14:66666666::::head 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: split 2/5 -> 14:66666666::::head 14:99999999::::head 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: split 3/5 -> 14:99999999::::head 14:cccccccc::::head 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: split 4/5 -> 14:cccccccc::::head MAX 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (2903 ms) 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] 7 tests from LibRadosList (140184 ms total) 2024-03-24T11:44:42.956 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1193 ms) 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset foo1,foo2,foo3 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo1 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo2 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo3 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset foo1,foo4,foo5 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo4 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo5 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo1 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset foo6,foo7 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo7 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: foo6 2024-03-24T11:44:42.957 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns1:foo4 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns1:foo5 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns2:foo7 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns2:foo6 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: ns1:foo1 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: :foo1 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: :foo2 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: :foo3 2024-03-24T11:44:42.958 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (28 ms) 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]': finished 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: from='client.55865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:42.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:42.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:42.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]': finished 2024-03-24T11:44:42.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: from='client.55865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:42.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T11:44:42.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:42.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:42 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi114-82498-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-25"}]': finished 2024-03-24T11:44:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: from='client.55865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi114-82433-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T11:44:43.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:43.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:42 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:43 smithi119 ceph-mon[33714]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T11:44:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:43 smithi119 ceph-mon[33714]: pgmap v295: 460 pgs: 8 unknown, 452 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[30312]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T11:44:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[30312]: pgmap v295: 460 pgs: 8 unknown, 452 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[42466]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T11:44:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]: dispatch 2024-03-24T11:44:44.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:43 smithi114 ceph-mon[42466]: pgmap v295: 460 pgs: 8 unknown, 452 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:44.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:44:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:44:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T11:44:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T11:44:44.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:44.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:44 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2195299602' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi114-82405-35"}]': finished 2024-03-24T11:44:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T11:44:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:45.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:44 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:46.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:46.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:46.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T11:44:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: pgmap v298: 420 pgs: 32 unknown, 388 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:45 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: pgmap v298: 420 pgs: 32 unknown, 388 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi114-82405-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3626483014' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: pgmap v298: 420 pgs: 32 unknown, 388 active+clean; 584 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]: dispatch 2024-03-24T11:44:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:45 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:46 smithi119 ceph-mon[33714]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:46 smithi119 ceph-mon[33714]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T11:44:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:46 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[30312]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[30312]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T11:44:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:47.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[42466]: from='client.55859 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi114-82498-2"}]': finished 2024-03-24T11:44:47.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[42466]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T11:44:47.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:47.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:46 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:48.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: pgmap v300: 388 pgs: 388 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:44:48.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T11:44:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3103900658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: from='client.55877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:48.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:47 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: pgmap v300: 388 pgs: 388 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:44:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3103900658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: from='client.55877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: pgmap v300: 388 pgs: 388 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi114-82405-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:48.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3103900658' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: from='client.55877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:47 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:48.798 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-24T11:44:48.798 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 1 0 2024-03-24T11:44:48.798 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 10 0 2024-03-24T11:44:48.798 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 13 0 2024-03-24T11:44:48.798 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 7 0 2024-03-24T11:44:48.798 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 14 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 0 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 15 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 11 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 5 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 8 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 6 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 3 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 4 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 12 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 9 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 2 0 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (41 ms) 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] 3 tests from LibRadosListEC (1262 ms total) 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 2024-03-24T11:44:48.799 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-24T11:44:48.800 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-24T11:44:48.800 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3031 ms) 2024-03-24T11:44:48.800 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] 1 test from LibRadosListNP (3031 ms total) 2024-03-24T11:44:48.800 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: 2024-03-24T11:44:48.800 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [----------] Global test environment tear-down 2024-03-24T11:44:48.800 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [==========] 11 tests from 3 test suites ran. (160917 ms total) 2024-03-24T11:44:48.800 INFO:tasks.workunit.client.0.smithi114.stdout: api_list: [ PASSED ] 11 tests. 2024-03-24T11:44:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]': finished 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.55877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]: dispatch 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]: dispatch 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/24012626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.55883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3103900658' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='client.55877 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-24T11:44:49.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]': finished 2024-03-24T11:44:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.55877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]: dispatch 2024-03-24T11:44:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T11:44:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]: dispatch 2024-03-24T11:44:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/24012626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.55883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3103900658' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-24T11:44:49.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='client.55877 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-24T11:44:49.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-27"}]': finished 2024-03-24T11:44:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.55877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82498-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:49.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]: dispatch 2024-03-24T11:44:49.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T11:44:49.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]: dispatch 2024-03-24T11:44:49.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/24012626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:49.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.55883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:49.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3103900658' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-24T11:44:49.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='client.55877 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-24T11:44:49.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: pgmap v303: 428 pgs: 72 unknown, 356 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-24T11:44:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]': finished 2024-03-24T11:44:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: from='client.55883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: from='client.55877 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-24T11:44:50.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T11:44:50.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:50.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:49 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: pgmap v303: 428 pgs: 72 unknown, 356 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-24T11:44:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]': finished 2024-03-24T11:44:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: from='client.55883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: from='client.55877 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: pgmap v303: 428 pgs: 72 unknown, 356 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-27", "mode": "writeback"}]': finished 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: from='client.55883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi114-82433-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: from='client.55877 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi114-82498-3","pool2":"test-rados-api-smithi114-82498-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T11:44:50.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:50.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:49 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:50 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:50 smithi119 ceph-mon[33714]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T11:44:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:50 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:51.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:51.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:50 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:51.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:51.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[30312]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T11:44:51.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:51.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:51.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:51.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:51.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:51.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[42466]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T11:44:51.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3685659896' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:51.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]: dispatch 2024-03-24T11:44:51.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:51.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:50 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:44:52.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: pgmap v306: 356 pgs: 356 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2247661180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.55889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: pgmap v306: 356 pgs: 356 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2247661180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.55889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: pgmap v306: 356 pgs: 356 active+clean; 584 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.55871 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi114-82405-36"}]': finished 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2247661180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.55889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:52.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:52.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]': finished 2024-03-24T11:44:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: from='client.55889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T11:44:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi114-82405-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:53 smithi119 ceph-mon[33714]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T11:44:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]': finished 2024-03-24T11:44:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: from='client.55889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi114-82405-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[30312]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-27"}]': finished 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: from='client.55889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi114-82433-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi114-82405-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:53.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:53 smithi114 ceph-mon[42466]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T11:44:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: pgmap v309: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:44:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:54.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi114-82405-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:54.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:54.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T11:44:54.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:54.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:54 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:54.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: pgmap v309: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:44:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi114-82405-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T11:44:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:54.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: pgmap v309: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:44:54.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi114-82405-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T11:44:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:44:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:44:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:44:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:44:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[30312]: pgmap v312: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[30312]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T11:44:56.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:56.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[42466]: pgmap v312: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:56.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:56.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[42466]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T11:44:56.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:55 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:55 smithi119 ceph-mon[33714]: pgmap v312: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:44:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:55 smithi119 ceph-mon[33714]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T11:44:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:55 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:44:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:44:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[30312]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T11:44:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:57.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[30312]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T11:44:57.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:44:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[42466]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T11:44:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[42466]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T11:44:57.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:44:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:56 smithi119 ceph-mon[33714]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T11:44:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:56 smithi119 ceph-mon[33714]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T11:44:57.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]: dispatch 2024-03-24T11:44:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: pgmap v315: 364 pgs: 8 creating+peering, 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:44:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: pgmap v315: 364 pgs: 8 creating+peering, 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:58.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:57 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: pgmap v315: 364 pgs: 8 creating+peering, 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:44:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:44:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/810717592' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi114-82405-37"}]': finished 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:58.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:57 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:44:59.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:44:59.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]': finished 2024-03-24T11:44:59.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:59.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T11:44:59.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]: dispatch 2024-03-24T11:44:59.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]: dispatch 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[30312]: pgmap v319: 356 pgs: 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]': finished 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T11:44:59.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]: dispatch 2024-03-24T11:44:59.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:44:59.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:59.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]: dispatch 2024-03-24T11:44:59.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:44:59.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:44:59 smithi114 ceph-mon[42466]: pgmap v319: 356 pgs: 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-24T11:45:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:45:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-29"}]': finished 2024-03-24T11:45:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi114-82405-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T11:45:00.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]: dispatch 2024-03-24T11:45:00.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]: dispatch 2024-03-24T11:45:00.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:00.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]: dispatch 2024-03-24T11:45:00.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:00.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:44:59 smithi119 ceph-mon[33714]: pgmap v319: 356 pgs: 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-24T11:45:00.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:00.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:45:00.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]': finished 2024-03-24T11:45:00.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T11:45:00.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1", "id": [0, 3]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.5", "id": [0, 2]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1f", "id": [0, 7]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.7", "id": [2, 6]}]: dispatch 2024-03-24T11:45:00.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.1a", "id": [2, 6]}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]': finished 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1", "id": [0, 3]}]: dispatch 2024-03-24T11:45:00.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.5", "id": [0, 2]}]: dispatch 2024-03-24T11:45:00.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1f", "id": [0, 7]}]: dispatch 2024-03-24T11:45:00.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.7", "id": [2, 6]}]: dispatch 2024-03-24T11:45:00.992 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:00 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.1a", "id": [2, 6]}]: dispatch 2024-03-24T11:45:01.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:01.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3599433960' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-46"}]': finished 2024-03-24T11:45:01.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-29", "mode": "writeback"}]': finished 2024-03-24T11:45:01.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1", "id": [0, 3]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.5", "id": [0, 2]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1f", "id": [0, 7]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.7", "id": [2, 6]}]: dispatch 2024-03-24T11:45:01.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:00 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.1a", "id": [2, 6]}]: dispatch 2024-03-24T11:45:01.955 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1", "id": [0, 3]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.5", "id": [0, 2]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1f", "id": [0, 7]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.7", "id": [2, 6]}]': finished 2024-03-24T11:45:01.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.1a", "id": [2, 6]}]': finished 2024-03-24T11:45:01.957 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.957 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T11:45:01.957 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.957 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:01 smithi119 ceph-mon[33714]: pgmap v322: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:01.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:01.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:01.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-24T11:45:01.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]': finished 2024-03-24T11:45:01.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-03-24T11:45:01.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1", "id": [0, 3]}]': finished 2024-03-24T11:45:01.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.5", "id": [0, 2]}]': finished 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1f", "id": [0, 7]}]': finished 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.7", "id": [2, 6]}]': finished 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.1a", "id": [2, 6]}]': finished 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[42466]: pgmap v322: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi114-82405-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi114-82433-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-24T11:45:01.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]': finished 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1", "id": [0, 3]}]': finished 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.5", "id": [0, 2]}]': finished 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "97.1f", "id": [0, 7]}]': finished 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.7", "id": [2, 6]}]': finished 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "224.1a", "id": [2, 6]}]': finished 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:01.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:01 smithi114 ceph-mon[30312]: pgmap v322: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[30312]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T11:45:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:02.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:02.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:02.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[42466]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T11:45:02.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:02.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:02 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:02 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:02 smithi119 ceph-mon[33714]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T11:45:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:03.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:04.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: pgmap v325: 364 pgs: 8 unknown, 8 peering, 348 active+clean; 8.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:04.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:45:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: pgmap v325: 364 pgs: 8 unknown, 8 peering, 348 active+clean; 8.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-24T11:45:04.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:04.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi114-82433-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:45:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T11:45:04.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]: dispatch 2024-03-24T11:45:04.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: pgmap v325: 364 pgs: 8 unknown, 8 peering, 348 active+clean; 8.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-24T11:45:04.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:04.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:04.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:45:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:45:04.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[42466]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:45:04.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:04.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:04.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]': finished 2024-03-24T11:45:04.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[42466]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[30312]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]': finished 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[30312]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:04 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:05.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:04 smithi119 ceph-mon[33714]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:45:05.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:04 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:05.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:04 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:05.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-29"}]': finished 2024-03-24T11:45:05.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:04 smithi119 ceph-mon[33714]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T11:45:05.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/474078755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:05.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:04 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]: dispatch 2024-03-24T11:45:06.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:06.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T11:45:06.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: pgmap v328: 324 pgs: 6 peering, 318 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:06.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:05 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: pgmap v328: 324 pgs: 6 peering, 318 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.55901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi114-82405-38"}]': finished 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: pgmap v328: 324 pgs: 6 peering, 318 active+clean; 4.6 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:05 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T11:45:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:07 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T11:45:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi114-82405-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1534903319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:07 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: pgmap v330: 356 pgs: 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 17 B/s, 8 objects/s recovering 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:08.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:08 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: pgmap v330: 356 pgs: 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 17 B/s, 8 objects/s recovering 2024-03-24T11:45:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T11:45:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: pgmap v330: 356 pgs: 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 17 B/s, 8 objects/s recovering 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi114-82433-47"}]': finished 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi114-82405-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi114-82433-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:08.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:08.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:08 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]': finished 2024-03-24T11:45:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]: dispatch 2024-03-24T11:45:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[30312]: pgmap v334: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s; 17 B/s, 8 objects/s recovering 2024-03-24T11:45:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[30312]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T11:45:09.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]: dispatch 2024-03-24T11:45:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]': finished 2024-03-24T11:45:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]: dispatch 2024-03-24T11:45:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[42466]: pgmap v334: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s; 17 B/s, 8 objects/s recovering 2024-03-24T11:45:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[42466]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T11:45:09.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]: dispatch 2024-03-24T11:45:10.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-31"}]': finished 2024-03-24T11:45:10.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]: dispatch 2024-03-24T11:45:10.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:09 smithi119 ceph-mon[33714]: pgmap v334: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s; 17 B/s, 8 objects/s recovering 2024-03-24T11:45:10.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:09 smithi119 ceph-mon[33714]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T11:45:10.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]: dispatch 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]': finished 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:10.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]': finished 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:11.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:11.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi114-82433-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:11.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-31", "mode": "writeback"}]': finished 2024-03-24T11:45:11.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T11:45:11.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:11.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:11.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:11.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:11.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[30312]: pgmap v337: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:11.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:11 smithi114 ceph-mon[42466]: pgmap v337: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:12.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3679008531' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]: dispatch 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]: dispatch 2024-03-24T11:45:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:11 smithi119 ceph-mon[33714]: pgmap v337: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:13.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]': finished 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:13.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:12 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:13.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:13.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:13.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]': finished 2024-03-24T11:45:13.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:13.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T11:45:13.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:13.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:13.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:13.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-31"}]': finished 2024-03-24T11:45:13.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.55913 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi114-82405-39"}]': finished 2024-03-24T11:45:13.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T11:45:13.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:13.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:13.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:13.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:13.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:12 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T11:45:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: pgmap v340: 324 pgs: 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:13 smithi119 ceph-mon[33714]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: pgmap v340: 324 pgs: 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[42466]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi114-82405-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2837094392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]: dispatch 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: pgmap v340: 324 pgs: 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:14.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:13 smithi114 ceph-mon[30312]: from='client.55919 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi114-82433-48"}]': finished 2024-03-24T11:45:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:45:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:45:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:45:15.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:15.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:15.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:15 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:15.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi114-82405-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi114-82433-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:15.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:15 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:16.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:16 smithi119 ceph-mon[33714]: pgmap v343: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:16.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:16 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:16.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:16.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:16 smithi119 ceph-mon[33714]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T11:45:16.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:16 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[30312]: pgmap v343: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[30312]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T11:45:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[42466]: pgmap v343: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:16.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:16.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:16.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[42466]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T11:45:16.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:16 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:18.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:18.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]': finished 2024-03-24T11:45:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T11:45:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]: dispatch 2024-03-24T11:45:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]: dispatch 2024-03-24T11:45:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:18.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:17 smithi119 ceph-mon[33714]: pgmap v346: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]': finished 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]: dispatch 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]: dispatch 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[30312]: pgmap v346: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi114-82433-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-33"}]': finished 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]: dispatch 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]: dispatch 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:18.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:17 smithi114 ceph-mon[42466]: pgmap v346: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]': finished 2024-03-24T11:45:19.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:19.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T11:45:19.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:19.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:19.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:19.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:19.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:18 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]': finished 2024-03-24T11:45:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T11:45:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-33", "mode": "writeback"}]': finished 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:19.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T11:45:19.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2150472961' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:19.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]: dispatch 2024-03-24T11:45:19.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:19.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:19.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:18 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:20.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:20.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: pgmap v349: 356 pgs: 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:20.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:19 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: pgmap v349: 356 pgs: 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:20.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi114-82405-40"}]': finished 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: pgmap v349: 356 pgs: 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:20.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:19 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:21.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:21.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:21.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]': finished 2024-03-24T11:45:21.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:21.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:21.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T11:45:21.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:21.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:21.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:21.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:20 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]': finished 2024-03-24T11:45:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-33"}]': finished 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi114-82405-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2043181744' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:21.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:21.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:20 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]: dispatch 2024-03-24T11:45:22.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:45:22.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: pgmap v351: 356 pgs: 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:22.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:22.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T11:45:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:21 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:22.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: pgmap v351: 356 pgs: 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:22.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:22.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: pgmap v351: 356 pgs: 356 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: from='client.55931 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi114-82433-49"}]': finished 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:22.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:22.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:21 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:23.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:23.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:23.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:23.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T11:45:23.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:23.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:23.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:23.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:22 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T11:45:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi114-82405-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi114-82433-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:22 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:24.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:23 smithi119 ceph-mon[33714]: pgmap v354: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:24.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:23 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:24.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:24.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:23 smithi119 ceph-mon[33714]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T11:45:24.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:23 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[30312]: pgmap v354: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[30312]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T11:45:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[42466]: pgmap v354: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:24.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:24.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:24.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[42466]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T11:45:24.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:23 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:45:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:45:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[30312]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[42466]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:24 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:24 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi114-82433-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:25.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:25.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:24 smithi119 ceph-mon[33714]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T11:45:25.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:25.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:25.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:24 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: pgmap v357: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]: dispatch 2024-03-24T11:45:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]: dispatch 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: pgmap v357: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]: dispatch 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]: dispatch 2024-03-24T11:45:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:25 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: pgmap v357: 364 pgs: 40 unknown, 324 active+clean; 4.6 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]: dispatch 2024-03-24T11:45:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T11:45:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3102063907' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]: dispatch 2024-03-24T11:45:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:25 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]: dispatch 2024-03-24T11:45:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]': finished 2024-03-24T11:45:27.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]': finished 2024-03-24T11:45:27.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:27.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:26 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-35", "mode": "writeback"}]': finished 2024-03-24T11:45:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.55937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi114-82405-41"}]': finished 2024-03-24T11:45:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T11:45:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:27.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:27.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:27.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:27.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:26 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: pgmap v360: 356 pgs: 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:28 smithi119 ceph-mon[33714]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: pgmap v360: 356 pgs: 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[30312]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: pgmap v360: 356 pgs: 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi114-82405-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:28.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T11:45:28.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1703703372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:28.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:28.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]: dispatch 2024-03-24T11:45:28.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:28.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:28 smithi114 ceph-mon[42466]: from='client.55943 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi114-82433-50"}]': finished 2024-03-24T11:45:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T11:45:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/26017243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.55955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:29.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:29.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T11:45:29.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/26017243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:29.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.55955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:29.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:29.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi114-82405-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/26017243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.55955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:29.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[30312]: pgmap v363: 356 pgs: 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[30312]: from='client.55955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[30312]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T11:45:30.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[42466]: pgmap v363: 356 pgs: 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[42466]: from='client.55955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:30 smithi114 ceph-mon[42466]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T11:45:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:30 smithi119 ceph-mon[33714]: pgmap v363: 356 pgs: 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:30 smithi119 ceph-mon[33714]: from='client.55955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi114-82433-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:30 smithi119 ceph-mon[33714]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T11:45:32.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:45:32.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: pgmap v366: 396 pgs: 40 unknown, 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:32.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:31 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: pgmap v366: 396 pgs: 40 unknown, 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:32.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:32.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: pgmap v366: 396 pgs: 40 unknown, 356 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:31 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:32.832 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7279 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6715 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7022 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6983 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7019 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7039 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7036 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7032 ms) 2024-03-24T11:45:32.833 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6912 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7049 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6917 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7050 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6975 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7018 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7044 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7023 ms) 2024-03-24T11:45:32.834 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112114 ms total) 2024-03-24T11:45:32.835 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: 2024-03-24T11:45:32.835 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [----------] Global test environment tear-down 2024-03-24T11:45:32.835 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (204999 ms total) 2024-03-24T11:45:32.835 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-24T11:45:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[30312]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:33.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:33 smithi114 ceph-mon[42466]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1771384253' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]: dispatch 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]: dispatch 2024-03-24T11:45:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:33.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:33.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:33.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:33.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:33.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:33 smithi119 ceph-mon[33714]: from='client.55949 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi114-82405-42"}]': finished 2024-03-24T11:45:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[42466]: pgmap v369: 356 pgs: 356 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T11:45:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[42466]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T11:45:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[42466]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[30312]: pgmap v369: 356 pgs: 356 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[30312]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-mon[30312]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T11:45:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:34 smithi119 ceph-mon[33714]: pgmap v369: 356 pgs: 356 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T11:45:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:34 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-35"}]': finished 2024-03-24T11:45:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:34 smithi119 ceph-mon[33714]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T11:45:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:34 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi114-82433-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:34 smithi119 ceph-mon[33714]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T11:45:34.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:45:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:45:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:45:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[30312]: pgmap v372: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:45:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[30312]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T11:45:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:36.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[42466]: pgmap v372: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:45:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[42466]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T11:45:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:35 smithi119 ceph-mon[33714]: pgmap v372: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:45:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:35 smithi119 ceph-mon[33714]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T11:45:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[30312]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T11:45:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[42466]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:37.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:36 smithi119 ceph-mon[33714]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T11:45:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:36 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: pgmap v375: 356 pgs: 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]': finished 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]: dispatch 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]: dispatch 2024-03-24T11:45:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: pgmap v375: 356 pgs: 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]': finished 2024-03-24T11:45:38.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:38.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T11:45:38.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]: dispatch 2024-03-24T11:45:38.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]: dispatch 2024-03-24T11:45:38.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:38.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:38.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: pgmap v375: 356 pgs: 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:45:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T11:45:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/254188369' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]: dispatch 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-37"}]': finished 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.55961 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi114-82433-52"}]': finished 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]: dispatch 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]: dispatch 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]': finished 2024-03-24T11:45:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[30312]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T11:45:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi114-82433-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]': finished 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[42466]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi114-82433-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:38 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-37", "mode": "writeback"}]': finished 2024-03-24T11:45:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:38 smithi119 ceph-mon[33714]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T11:45:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi114-82433-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:40.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:39 smithi114 ceph-mon[30312]: pgmap v379: 356 pgs: 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:45:40.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:39 smithi114 ceph-mon[42466]: pgmap v379: 356 pgs: 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:45:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:39 smithi119 ceph-mon[33714]: pgmap v379: 356 pgs: 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:45:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[30312]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T11:45:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi114-82433-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]': finished 2024-03-24T11:45:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[30312]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T11:45:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[42466]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T11:45:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi114-82433-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]': finished 2024-03-24T11:45:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:40 smithi114 ceph-mon[42466]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T11:45:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:40 smithi119 ceph-mon[33714]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T11:45:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]: dispatch 2024-03-24T11:45:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:40 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi114-82433-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-37"}]': finished 2024-03-24T11:45:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:40 smithi119 ceph-mon[33714]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T11:45:41.994 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:45:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:45:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:41 smithi119 ceph-mon[33714]: pgmap v382: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:41 smithi119 ceph-mon[33714]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T11:45:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:41 smithi114 ceph-mon[30312]: pgmap v382: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:41 smithi114 ceph-mon[30312]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T11:45:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:41 smithi114 ceph-mon[42466]: pgmap v382: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:41 smithi114 ceph-mon[42466]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T11:45:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:42 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:42 smithi119 ceph-mon[33714]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T11:45:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[30312]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T11:45:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[42466]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T11:45:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:43 smithi119 ceph-mon[33714]: pgmap v385: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:45:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:43 smithi119 ceph-mon[33714]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T11:45:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[30312]: pgmap v385: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:45:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[30312]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T11:45:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[42466]: pgmap v385: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:45:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[42466]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T11:45:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]: dispatch 2024-03-24T11:45:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:44.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:44 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:44.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:45:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:45:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:45:46.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:46.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T11:45:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: pgmap v388: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:45:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:46.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi114-82433-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T11:45:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:46.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: pgmap v388: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi114-82433-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/496810035' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi114-82433-53"}]': finished 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: pgmap v388: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:46.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi114-82433-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:47.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]': finished 2024-03-24T11:45:47.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi114-82433-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:47.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]: dispatch 2024-03-24T11:45:47.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:47 smithi119 ceph-mon[33714]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T11:45:47.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi114-82433-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:47.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]: dispatch 2024-03-24T11:45:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]': finished 2024-03-24T11:45:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi114-82433-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]: dispatch 2024-03-24T11:45:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[42466]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T11:45:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi114-82433-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]: dispatch 2024-03-24T11:45:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-39"}]': finished 2024-03-24T11:45:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi114-82433-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]: dispatch 2024-03-24T11:45:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[30312]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T11:45:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi114-82433-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]: dispatch 2024-03-24T11:45:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]': finished 2024-03-24T11:45:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: pgmap v391: 356 pgs: 356 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T11:45:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi114-82433-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T11:45:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]': finished 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: pgmap v391: 356 pgs: 356 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi114-82433-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:45:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-39", "mode": "writeback"}]': finished 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: pgmap v391: 356 pgs: 356 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi114-82433-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:48.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]: dispatch 2024-03-24T11:45:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]': finished 2024-03-24T11:45:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[30312]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T11:45:49.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:49.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:49.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:49.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]': finished 2024-03-24T11:45:49.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:49 smithi114 ceph-mon[42466]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T11:45:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:45:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:49 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:45:49.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:49 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-39"}]': finished 2024-03-24T11:45:49.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:49 smithi119 ceph-mon[33714]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T11:45:49.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-24T11:45:49.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-24T11:45:49.777 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: seed 82677 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (318 ms) 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (15259 ms) 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (127 ms) 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (15704 ms total) 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (8120 ms) 2024-03-24T11:45:49.778 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (9075 ms) 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9990 ms) 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: my_snaps [3] 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: my_snaps [4,3] 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: promoting some heads 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: waiting for scrubs... 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: done waiting 2024-03-24T11:45:49.779 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46309 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9712 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7997 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8024 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8040 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10050 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8918 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10304 ms) 2024-03-24T11:45:49.780 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (11708 ms) 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7985 ms) 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8280 ms) 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12818 ms) 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7905 ms) 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-24T11:45:49.781 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8034 ms) 2024-03-24T11:45:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:50 smithi114 ceph-mon[30312]: pgmap v394: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:50 smithi114 ceph-mon[30312]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T11:45:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:50 smithi114 ceph-mon[42466]: pgmap v394: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:50 smithi114 ceph-mon[42466]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T11:45:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:50 smithi119 ceph-mon[33714]: pgmap v394: 364 pgs: 8 unknown, 356 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:50 smithi119 ceph-mon[33714]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T11:45:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:52.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:51 smithi119 ceph-mon[33714]: pgmap v397: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:51 smithi119 ceph-mon[33714]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T11:45:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:52.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:52.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:45:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:45:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[30312]: pgmap v397: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[30312]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T11:45:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:52.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[42466]: pgmap v397: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[42466]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T11:45:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]: dispatch 2024-03-24T11:45:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:52.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:45:53.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:53.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:45:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T11:45:53.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:45:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2906170390' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi114-82433-54"}]': finished 2024-03-24T11:45:53.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:45:53.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:45:53.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:45:53.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi114-82433-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:45:53.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:45:53.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T11:45:53.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:45:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:54 smithi119 ceph-mon[33714]: pgmap v399: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 974 KiB/s wr, 1 op/s 2024-03-24T11:45:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:54.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:54 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:54.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:54 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:45:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[30312]: pgmap v399: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 974 KiB/s wr, 1 op/s 2024-03-24T11:45:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:45:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[42466]: pgmap v399: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 974 KiB/s wr, 1 op/s 2024-03-24T11:45:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:54.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:45:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:45:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:45:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:45:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:45:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T11:45:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:45:55.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:45:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:45:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T11:45:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:45:55.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:45:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:45:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T11:45:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:45:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:45:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:45:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi114-82433-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:45:55.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:45:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:56 smithi119 ceph-mon[33714]: pgmap v402: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-24T11:45:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:56 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:45:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:56 smithi119 ceph-mon[33714]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T11:45:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:56 smithi114 ceph-mon[30312]: pgmap v402: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-24T11:45:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:56 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:45:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:56 smithi114 ceph-mon[30312]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T11:45:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:56 smithi114 ceph-mon[42466]: pgmap v402: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-24T11:45:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:56 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:45:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:56 smithi114 ceph-mon[42466]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T11:45:58.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: pgmap v405: 364 pgs: 364 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:58.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T11:45:58.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:58.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:58.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:58.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:58.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]: dispatch 2024-03-24T11:45:58.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]: dispatch 2024-03-24T11:45:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: pgmap v405: 364 pgs: 364 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T11:45:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: pgmap v405: 364 pgs: 364 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]: dispatch 2024-03-24T11:45:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]: dispatch 2024-03-24T11:45:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:45:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]': finished 2024-03-24T11:45:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[30312]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T11:45:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:59.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:59.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:45:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]': finished 2024-03-24T11:45:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[42466]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T11:45:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:59.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:58 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:58 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:45:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-41"}]': finished 2024-03-24T11:45:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:58 smithi119 ceph-mon[33714]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T11:45:59.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2723678081' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:45:59.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:58 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]: dispatch 2024-03-24T11:46:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: pgmap v408: 356 pgs: 356 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:46:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T11:46:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: pgmap v408: 356 pgs: 356 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:00.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:45:59 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:00.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: pgmap v408: 356 pgs: 356 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:00.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:00.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: from='client.55979 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi114-82433-55"}]': finished 2024-03-24T11:46:00.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T11:46:00.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:00.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:00.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:45:59 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:01.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:01.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[42466]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T11:46:01.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:01.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[30312]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:01.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi114-82433-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-mon[33714]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T11:46:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:01.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:01.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:01.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:01.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:02.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:46:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: pgmap v411: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi114-82677-6","var": "pg_num","format": "json"}]: dispatch 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-24T11:46:02.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: pgmap v411: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi114-82677-6","var": "pg_num","format": "json"}]: dispatch 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-24T11:46:02.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-24T11:46:02.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: pgmap v411: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:02.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi114-82677-6","var": "pg_num","format": "json"}]: dispatch 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi114-82433-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-24T11:46:02.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-24T11:46:03.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:03.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]': finished 2024-03-24T11:46:03.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[30312]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T11:46:03.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:03.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:03.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:03.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]': finished 2024-03-24T11:46:03.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[42466]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T11:46:03.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:03.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_count","val": "8"}]': finished 2024-03-24T11:46:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:03 smithi119 ceph-mon[33714]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T11:46:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:03.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: pgmap v414: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:46:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T11:46:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-24T11:46:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: pgmap v414: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-24T11:46:04.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: pgmap v414: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:46:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T11:46:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-24T11:46:04.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:04.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-24T11:46:04.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:04 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:04.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:46:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:46:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: ok, hit_set contains 244:602f83fe:::foo:head 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9040 ms) 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg_num = 32 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 0 ls 1711280765,0 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 1 ls 1711280765,0 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 2 ls 1711280765,0 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 3 ls 1711280765,0 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 4 ls 1711280765,0 2024-03-24T11:46:05.060 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 5 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 6 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 7 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 8 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 9 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 10 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 11 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 12 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 13 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 14 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 15 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 16 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 17 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 18 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 19 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 20 ls 1711280765,0 2024-03-24T11:46:05.061 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 21 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 22 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 23 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 24 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 25 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 26 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 27 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 28 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 29 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 30 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg 31 ls 1711280765,0 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: pg_num = 32 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6cac518f:::0:head 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:02547ec2:::1:head 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f905c69b:::2:head 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:cfc208b3:::3:head 2024-03-24T11:46:05.062 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d83876eb:::4:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b29083e3:::5:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c4fdafeb:::6:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:5c6b0b28:::7:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:bd63b0f1:::8:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:e960b815:::9:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:52ea6a34:::10:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:89d3ae78:::11:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:de5d7c5f:::12:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:566253c9:::13:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:62a1935d:::14:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:863748b0:::15:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:3958e169:::16:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:4d4dabf9:::17:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:8391935d:::18:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:28883081:::19:head 2024-03-24T11:46:05.063 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:69259c59:::20:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:4bdb80b7:::21:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a11c5d71:::22:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:271af37b:::23:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:95b121be:::24:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:58d1031b:::25:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:0a050783:::26:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c709704c:::27:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:cbe56eaf:::28:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:86b4b162:::29:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:70d89383:::30:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:dd450c7c:::31:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6d5729b1:::32:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c388f3fb:::33:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:56cfea31:::34:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:9dbc1bf7:::35:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:40b74ccd:::36:head 2024-03-24T11:46:05.064 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:4d5aaf42:::37:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:920f362c:::38:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6cc53222:::39:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:9cad833f:::40:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:1ea84d41:::41:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c4480ef6:::42:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a694361e:::43:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d1bd33e9:::44:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:ddc2cd5d:::45:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:2b782207:::46:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:7b187fca:::47:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:90ecdf6f:::48:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a5ed95fe:::49:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:ea0eaa55:::50:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f33ef17b:::51:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a0d1b2f6:::52:head 2024-03-24T11:46:05.065 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:60c5229e:::53:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:edcbc575:::54:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:102cf253:::55:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:efb7fb0b:::56:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:50d0a326:::57:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d4dc5daf:::58:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:3a130462:::59:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:ec87ed71:::60:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d5bc9454:::61:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:3ddfe313:::62:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:7c2816b9:::63:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:47e00e4d:::64:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c6410c18:::65:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b48ed237:::66:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:cd63ad31:::67:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b179e92b:::68:head 2024-03-24T11:46:05.066 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:0d9f741a:::69:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6d3352ae:::70:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c6d5c19e:::71:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:bc4729c3:::72:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:77e930b9:::73:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:0abeecfd:::74:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b7c37e15:::75:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b6378398:::76:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:02bd68de:::77:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:cc795d2d:::78:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:630d4fea:::79:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:e0d29ef5:::80:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:fd6f13d2:::81:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:606461d5:::82:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:eadbdc43:::83:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:8761d0bb:::84:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:9ef0186f:::85:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:e0d41294:::86:head 2024-03-24T11:46:05.067 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:961de695:::87:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:1423148f:::88:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:633a8fa2:::89:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a8653809:::90:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:3dac8b33:::91:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:35aad435:::92:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f6dcc343:::93:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:dbbdad87:::94:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:1cb48ce0:::95:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:03cd461c:::96:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:17a4ea99:::97:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:9993c9a7:::98:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6394211c:::99:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:94c7ae57:::100:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6fdee5bb:::101:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:9a477fd1:::102:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:eb850916:::103:head 2024-03-24T11:46:05.068 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:affc56b9:::104:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b42dc814:::105:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f319f8f0:::106:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:9a40b9de:::107:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:8b524f28:::108:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:e3de589f:::109:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:90f90a5b:::110:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a7b4f1d7:::111:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:af51766e:::112:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b6f90bd1:::113:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:e0261208:::114:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c9569ef7:::115:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:61bebe50:::116:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:fe93412b:::117:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d3d38bee:::118:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:3100ba0c:::119:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d0560ada:::120:head 2024-03-24T11:46:05.069 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f0ea8b35:::121:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:766f231a:::122:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a07a2582:::123:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:bd7c6b3a:::124:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:fb2ddaff:::125:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:4408e1fe:::126:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:ee1df7a7:::127:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c3002909:::128:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:4f48ffa9:::129:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:edf38733:::130:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c08425c0:::131:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:5f902d98:::132:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:41ea2c93:::133:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:813cee13:::134:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:0131818d:::135:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:26ba5a85:::136:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:381b8a5a:::137:head 2024-03-24T11:46:05.070 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:28797e47:::138:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:bfca7f22:::139:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:36807075:::140:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:80b03975:::141:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:5c15709b:::142:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f39ea15e:::143:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:ea992956:::144:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:48887b1c:::145:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:9f24a9dd:::146:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:987f100b:::147:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d2dd3581:::148:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:7fed1808:::149:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c80b70e9:::150:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:85ed90f9:::151:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:36428b24:::152:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d044c34a:::153:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:7c18bf58:::154:head 2024-03-24T11:46:05.071 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d1c21232:::155:head 2024-03-24T11:46:05.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a7a3c575:::156:head 2024-03-24T11:46:05.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:87da0633:::157:head 2024-03-24T11:46:05.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d5ac3822:::158:head 2024-03-24T11:46:05.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:3f20522d:::159:head 2024-03-24T11:46:05.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6ca26563:::160:head 2024-03-24T11:46:05.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:532ce135:::161:head 2024-03-24T11:46:05.072 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:c78863e6:::162:head 2024-03-24T11:46:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: pgmap v417: 356 pgs: 356 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-24T11:46:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T11:46:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi114-82677-43","var": "pg_num","format": "json"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: pgmap v417: 356 pgs: 356 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi114-82677-43","var": "pg_num","format": "json"}]: dispatch 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:06.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]: dispatch 2024-03-24T11:46:06.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]: dispatch 2024-03-24T11:46:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: pgmap v417: 356 pgs: 356 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-24T11:46:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2205172596' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]: dispatch 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi114-82677-43","var": "pg_num","format": "json"}]: dispatch 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]: dispatch 2024-03-24T11:46:06.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]: dispatch 2024-03-24T11:46:07.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[30312]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:07.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]': finished 2024-03-24T11:46:07.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[30312]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T11:46:07.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:07.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:07.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi114-82433-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:07.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[42466]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:07.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]': finished 2024-03-24T11:46:07.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[42466]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T11:46:07.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:07.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:07.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi114-82433-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:06 smithi119 ceph-mon[33714]: from='client.55985 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi114-82433-56"}]': finished 2024-03-24T11:46:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-43"}]': finished 2024-03-24T11:46:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:06 smithi119 ceph-mon[33714]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T11:46:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:07.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi114-82433-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[30312]: pgmap v420: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 101 op/s 2024-03-24T11:46:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi114-82433-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[30312]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T11:46:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi114-82433-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[42466]: pgmap v420: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 101 op/s 2024-03-24T11:46:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi114-82433-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[42466]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T11:46:08.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi114-82433-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:08 smithi119 ceph-mon[33714]: pgmap v420: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 101 op/s 2024-03-24T11:46:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi114-82433-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:08 smithi119 ceph-mon[33714]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T11:46:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi114-82433-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T11:46:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi114-82433-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T11:46:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi114-82433-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:09.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T11:46:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi114-82433-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T11:46:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:09.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[30312]: pgmap v423: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[30312]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T11:46:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:46:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:46:10.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[42466]: pgmap v423: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:10.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:10.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[42466]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T11:46:10.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:46:10.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:46:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:10 smithi119 ceph-mon[33714]: pgmap v423: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:10 smithi119 ceph-mon[33714]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T11:46:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:46:10.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:46:12.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:46:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:46:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:11 smithi119 ceph-mon[33714]: pgmap v426: 364 pgs: 40 unknown, 324 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]': finished 2024-03-24T11:46:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:11 smithi119 ceph-mon[33714]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T11:46:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:46:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:12.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:11 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:46:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[30312]: pgmap v426: 364 pgs: 40 unknown, 324 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]': finished 2024-03-24T11:46:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[30312]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T11:46:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:46:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:12.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:46:12.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[42466]: pgmap v426: 364 pgs: 40 unknown, 324 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:12.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_count","val": "3"}]': finished 2024-03-24T11:46:12.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[42466]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T11:46:12.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:46:12.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:12.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:11 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:46:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]': finished 2024-03-24T11:46:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T11:46:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:13.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[30312]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]': finished 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:13.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:46:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:46:13.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:12 smithi114 ceph-mon[42466]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T11:46:13.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_period","val": "3"}]': finished 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]: dispatch 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2709571264' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi114-82433-57"}]': finished 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:46:13.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:12 smithi119 ceph-mon[33714]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T11:46:14.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: pgmap v429: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:14.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:46:14.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi114-82433-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi114-82433-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi114-82433-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: pgmap v429: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi114-82433-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-24T11:46:14.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi114-82433-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:14.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T11:46:14.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi114-82433-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: pgmap v429: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:46:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi114-82433-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-24T11:46:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi114-82433-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T11:46:14.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi114-82433-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:46:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:46:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:46:16.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:15 smithi114 ceph-mon[30312]: pgmap v432: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:16.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:15 smithi114 ceph-mon[30312]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T11:46:16.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:15 smithi114 ceph-mon[42466]: pgmap v432: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:16.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:15 smithi114 ceph-mon[42466]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T11:46:16.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:15 smithi119 ceph-mon[33714]: pgmap v432: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:16.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:15 smithi119 ceph-mon[33714]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T11:46:17.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi114-82433-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:17.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:16 smithi114 ceph-mon[30312]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T11:46:17.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi114-82433-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:17.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:16 smithi114 ceph-mon[42466]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T11:46:17.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi114-82433-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:17.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:16 smithi119 ceph-mon[33714]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T11:46:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:17 smithi114 ceph-mon[30312]: pgmap v435: 364 pgs: 8 creating+peering, 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:46:18.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:17 smithi114 ceph-mon[30312]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T11:46:18.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:17 smithi114 ceph-mon[42466]: pgmap v435: 364 pgs: 8 creating+peering, 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:46:18.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:17 smithi114 ceph-mon[42466]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T11:46:18.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:17 smithi119 ceph-mon[33714]: pgmap v435: 364 pgs: 8 creating+peering, 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:46:18.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:17 smithi119 ceph-mon[33714]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T11:46:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[42466]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T11:46:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:19.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:19.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[30312]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T11:46:19.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:19.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:19.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:19.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:18 smithi119 ceph-mon[33714]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T11:46:19.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:19.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:18 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:19.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[30312]: pgmap v438: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:46:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[30312]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T11:46:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[30312]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T11:46:20.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[42466]: pgmap v438: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:46:20.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[42466]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T11:46:20.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:20.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:20.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:20.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:19 smithi114 ceph-mon[42466]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T11:46:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:19 smithi119 ceph-mon[33714]: pgmap v438: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:46:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:19 smithi119 ceph-mon[33714]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T11:46:20.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:19 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:20.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]: dispatch 2024-03-24T11:46:20.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3716652055' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi114-82433-58"}]': finished 2024-03-24T11:46:20.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:19 smithi119 ceph-mon[33714]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T11:46:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:21.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:20 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:20 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:20 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:20 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:22.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:21 smithi119 ceph-mon[33714]: pgmap v441: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:22.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:21 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:21 smithi119 ceph-mon[33714]: osdmap e318: 8 total, 8 up, 8 in 2024-03-24T11:46:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:22.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:21 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:22.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:46:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:46:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[30312]: pgmap v441: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[30312]: osdmap e318: 8 total, 8 up, 8 in 2024-03-24T11:46:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[42466]: pgmap v441: 356 pgs: 356 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[42466]: osdmap e318: 8 total, 8 up, 8 in 2024-03-24T11:46:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:22.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:21 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:22 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:22 smithi114 ceph-mon[42466]: osdmap e319: 8 total, 8 up, 8 in 2024-03-24T11:46:23.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:22 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:23.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:22 smithi114 ceph-mon[30312]: osdmap e319: 8 total, 8 up, 8 in 2024-03-24T11:46:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:22 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:22 smithi119 ceph-mon[33714]: osdmap e319: 8 total, 8 up, 8 in 2024-03-24T11:46:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:23 smithi114 ceph-mon[42466]: pgmap v444: 356 pgs: 356 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:46:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:23 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:23 smithi114 ceph-mon[42466]: osdmap e320: 8 total, 8 up, 8 in 2024-03-24T11:46:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:23 smithi114 ceph-mon[30312]: pgmap v444: 356 pgs: 356 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:46:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:23 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:24.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:23 smithi114 ceph-mon[30312]: osdmap e320: 8 total, 8 up, 8 in 2024-03-24T11:46:24.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:23 smithi119 ceph-mon[33714]: pgmap v444: 356 pgs: 356 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:46:24.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:23 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi114-82433-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:24.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:23 smithi119 ceph-mon[33714]: osdmap e320: 8 total, 8 up, 8 in 2024-03-24T11:46:24.925 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:46:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:46:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:46:25.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:24 smithi114 ceph-mon[30312]: osdmap e321: 8 total, 8 up, 8 in 2024-03-24T11:46:25.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:24 smithi114 ceph-mon[42466]: osdmap e321: 8 total, 8 up, 8 in 2024-03-24T11:46:25.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:24 smithi119 ceph-mon[33714]: osdmap e321: 8 total, 8 up, 8 in 2024-03-24T11:46:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: pgmap v447: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:46:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: osdmap e322: 8 total, 8 up, 8 in 2024-03-24T11:46:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: pgmap v447: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: osdmap e322: 8 total, 8 up, 8 in 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:26.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]: dispatch 2024-03-24T11:46:26.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]: dispatch 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: pgmap v447: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: osdmap e322: 8 total, 8 up, 8 in 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]: dispatch 2024-03-24T11:46:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]: dispatch 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:7462ddf6:::o.RoundTripAppendPP (3251 ms) 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2773 ms) 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2967 ms) 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3017 ms) 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: setting pool EIO 2024-03-24T11:46:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: max_success 100, min_failed 101 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4013 ms) 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3017 ms) 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (131923 ms total) 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3017 ms) 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9287 ms) 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14710 ms) 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3015 ms) 2024-03-24T11:46:26.938 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30029 ms total) 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3257 ms) 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3257 ms total) 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13725 ms) 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6978 ms) 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7033 ms) 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7038 ms) 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-24T11:46:26.939 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3007 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6897 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7031 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7032 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7031 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7014 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7024 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7019 ms) 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-24T11:46:26.940 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7060 ms) 2024-03-24T11:46:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]': finished 2024-03-24T11:46:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[30312]: osdmap e323: 8 total, 8 up, 8 in 2024-03-24T11:46:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:27.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]': finished 2024-03-24T11:46:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[42466]: osdmap e323: 8 total, 8 up, 8 in 2024-03-24T11:46:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:27 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:27 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-45"}]': finished 2024-03-24T11:46:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:27 smithi119 ceph-mon[33714]: osdmap e323: 8 total, 8 up, 8 in 2024-03-24T11:46:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2102465189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:27 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]: dispatch 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: pgmap v450: 356 pgs: 356 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: osdmap e324: 8 total, 8 up, 8 in 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: osdmap e325: 8 total, 8 up, 8 in 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3293940110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: from='client.56009 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: pgmap v450: 356 pgs: 356 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: osdmap e324: 8 total, 8 up, 8 in 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: osdmap e325: 8 total, 8 up, 8 in 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3293940110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: from='client.56009 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: pgmap v450: 356 pgs: 356 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:46:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: from='client.56003 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi114-82433-59"}]': finished 2024-03-24T11:46:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: osdmap e324: 8 total, 8 up, 8 in 2024-03-24T11:46:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: osdmap e325: 8 total, 8 up, 8 in 2024-03-24T11:46:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3293940110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: from='client.56009 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:28.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[42466]: pgmap v453: 388 pgs: 64 unknown, 324 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[42466]: from='client.56009 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[42466]: osdmap e326: 8 total, 8 up, 8 in 2024-03-24T11:46:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[30312]: pgmap v453: 388 pgs: 64 unknown, 324 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[30312]: from='client.56009 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[30312]: osdmap e326: 8 total, 8 up, 8 in 2024-03-24T11:46:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:30.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:30.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:29 smithi119 ceph-mon[33714]: pgmap v453: 388 pgs: 64 unknown, 324 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:30.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:29 smithi119 ceph-mon[33714]: from='client.56009 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi114-82433-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:29 smithi119 ceph-mon[33714]: osdmap e326: 8 total, 8 up, 8 in 2024-03-24T11:46:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:30.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: osdmap e327: 8 total, 8 up, 8 in 2024-03-24T11:46:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi114-82433-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]': finished 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi114-82433-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: osdmap e328: 8 total, 8 up, 8 in 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]: dispatch 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi114-82433-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]: dispatch 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: osdmap e327: 8 total, 8 up, 8 in 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi114-82433-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:31.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]': finished 2024-03-24T11:46:31.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi114-82433-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:31.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: osdmap e328: 8 total, 8 up, 8 in 2024-03-24T11:46:31.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]: dispatch 2024-03-24T11:46:31.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi114-82433-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]: dispatch 2024-03-24T11:46:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: osdmap e327: 8 total, 8 up, 8 in 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi114-82433-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-47"}]': finished 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi114-82433-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: osdmap e328: 8 total, 8 up, 8 in 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]: dispatch 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi114-82433-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:31.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]: dispatch 2024-03-24T11:46:32.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:46:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:46:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[30312]: pgmap v456: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:46:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]': finished 2024-03-24T11:46:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[42466]: pgmap v456: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:46:32.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:32 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]': finished 2024-03-24T11:46:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:32 smithi119 ceph-mon[33714]: pgmap v456: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:32 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:32 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:46:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:32 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-47", "mode": "writeback"}]': finished 2024-03-24T11:46:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:46:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: osdmap e329: 8 total, 8 up, 8 in 2024-03-24T11:46:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:46:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi114-82433-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: osdmap e330: 8 total, 8 up, 8 in 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:46:33.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: osdmap e329: 8 total, 8 up, 8 in 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi114-82433-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: osdmap e330: 8 total, 8 up, 8 in 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:33.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:46:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: osdmap e329: 8 total, 8 up, 8 in 2024-03-24T11:46:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi114-82433-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: osdmap e330: 8 total, 8 up, 8 in 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:46:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[42466]: pgmap v459: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:46:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:46:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[42466]: osdmap e331: 8 total, 8 up, 8 in 2024-03-24T11:46:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[30312]: pgmap v459: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:46:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:46:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[30312]: osdmap e331: 8 total, 8 up, 8 in 2024-03-24T11:46:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:34.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:34 smithi119 ceph-mon[33714]: pgmap v459: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:46:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:34 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:46:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:34 smithi119 ceph-mon[33714]: osdmap e331: 8 total, 8 up, 8 in 2024-03-24T11:46:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:34.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:34 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:46:34.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:46:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:46:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:46:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:46:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:46:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[42466]: osdmap e332: 8 total, 8 up, 8 in 2024-03-24T11:46:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:46:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:46:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:35.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:46:35.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:46:35.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[30312]: osdmap e332: 8 total, 8 up, 8 in 2024-03-24T11:46:35.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:46:35.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:46:35.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:35 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:46:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:46:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:35 smithi119 ceph-mon[33714]: osdmap e332: 8 total, 8 up, 8 in 2024-03-24T11:46:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:46:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:46:35.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[42466]: pgmap v462: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:46:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:46:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[42466]: osdmap e333: 8 total, 8 up, 8 in 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[30312]: pgmap v462: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[30312]: osdmap e333: 8 total, 8 up, 8 in 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:46:36.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:46:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:36 smithi119 ceph-mon[33714]: pgmap v462: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:46:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:46:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:36 smithi119 ceph-mon[33714]: osdmap e333: 8 total, 8 up, 8 in 2024-03-24T11:46:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]: dispatch 2024-03-24T11:46:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:46:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:46:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: pgmap v465: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-24T11:46:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: osdmap e334: 8 total, 8 up, 8 in 2024-03-24T11:46:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: pgmap v465: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: osdmap e334: 8 total, 8 up, 8 in 2024-03-24T11:46:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:38.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:37 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: pgmap v465: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1400681921' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi114-82433-61"}]': finished 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: osdmap e334: 8 total, 8 up, 8 in 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:38.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:37 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-24T11:46:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[42466]: osdmap e335: 8 total, 8 up, 8 in 2024-03-24T11:46:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-24T11:46:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[30312]: osdmap e335: 8 total, 8 up, 8 in 2024-03-24T11:46:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-24T11:46:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:38 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:38 smithi119 ceph-mon[33714]: osdmap e335: 8 total, 8 up, 8 in 2024-03-24T11:46:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:38 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[42466]: pgmap v468: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:46:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[42466]: osdmap e336: 8 total, 8 up, 8 in 2024-03-24T11:46:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:40.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[30312]: pgmap v468: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:40.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:46:40.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:40.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[30312]: osdmap e336: 8 total, 8 up, 8 in 2024-03-24T11:46:40.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:39 smithi119 ceph-mon[33714]: pgmap v468: 356 pgs: 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:46:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:39 smithi119 ceph-mon[33714]: osdmap e336: 8 total, 8 up, 8 in 2024-03-24T11:46:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]': finished 2024-03-24T11:46:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[30312]: osdmap e337: 8 total, 8 up, 8 in 2024-03-24T11:46:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]': finished 2024-03-24T11:46:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[42466]: osdmap e337: 8 total, 8 up, 8 in 2024-03-24T11:46:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:41.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:40 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi114-82433-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]': finished 2024-03-24T11:46:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:40 smithi119 ceph-mon[33714]: osdmap e337: 8 total, 8 up, 8 in 2024-03-24T11:46:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-47"}]: dispatch 2024-03-24T11:46:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:41 smithi119 ceph-mon[33714]: pgmap v471: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:42.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:41 smithi119 ceph-mon[33714]: osdmap e338: 8 total, 8 up, 8 in 2024-03-24T11:46:42.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:46:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:46:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:41 smithi114 ceph-mon[42466]: pgmap v471: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:41 smithi114 ceph-mon[42466]: osdmap e338: 8 total, 8 up, 8 in 2024-03-24T11:46:42.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:41 smithi114 ceph-mon[30312]: pgmap v471: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:42.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:41 smithi114 ceph-mon[30312]: osdmap e338: 8 total, 8 up, 8 in 2024-03-24T11:46:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:42 smithi119 ceph-mon[33714]: osdmap e339: 8 total, 8 up, 8 in 2024-03-24T11:46:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:42 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:43.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:42 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[30312]: osdmap e339: 8 total, 8 up, 8 in 2024-03-24T11:46:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[42466]: osdmap e339: 8 total, 8 up, 8 in 2024-03-24T11:46:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:46:43.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:42 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: pgmap v474: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:46:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: osdmap e340: 8 total, 8 up, 8 in 2024-03-24T11:46:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:44.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: pgmap v474: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:46:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: osdmap e340: 8 total, 8 up, 8 in 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: pgmap v474: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: osdmap e340: 8 total, 8 up, 8 in 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/149784771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]: dispatch 2024-03-24T11:46:44.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:44.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:43 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:46:44.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:46:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:46:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:46:45.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:45.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:45.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: osdmap e341: 8 total, 8 up, 8 in 2024-03-24T11:46:45.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:45.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:45.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:45.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:45 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: osdmap e341: 8 total, 8 up, 8 in 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:45.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.56021 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi114-82433-62"}]': finished 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: osdmap e341: 8 total, 8 up, 8 in 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:45.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:45 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: pgmap v477: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:46:46.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]': finished 2024-03-24T11:46:46.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:46.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]: dispatch 2024-03-24T11:46:46.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: osdmap e342: 8 total, 8 up, 8 in 2024-03-24T11:46:46.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:46.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]: dispatch 2024-03-24T11:46:46.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:46 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: pgmap v477: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]': finished 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]: dispatch 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: osdmap e342: 8 total, 8 up, 8 in 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]: dispatch 2024-03-24T11:46:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: pgmap v477: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-49"}]': finished 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi114-82433-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]: dispatch 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: osdmap e342: 8 total, 8 up, 8 in 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]: dispatch 2024-03-24T11:46:46.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:46 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:47.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:47 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:46:47.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]': finished 2024-03-24T11:46:47.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:47 smithi119 ceph-mon[33714]: osdmap e343: 8 total, 8 up, 8 in 2024-03-24T11:46:47.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:47 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:47.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:47 smithi119 ceph-mon[33714]: osdmap e344: 8 total, 8 up, 8 in 2024-03-24T11:46:47.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:46:47.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]': finished 2024-03-24T11:46:47.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[42466]: osdmap e343: 8 total, 8 up, 8 in 2024-03-24T11:46:47.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:47.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[42466]: osdmap e344: 8 total, 8 up, 8 in 2024-03-24T11:46:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:46:47.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-49", "mode": "readproxy"}]': finished 2024-03-24T11:46:47.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[30312]: osdmap e343: 8 total, 8 up, 8 in 2024-03-24T11:46:47.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi114-82433-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:47.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:47 smithi114 ceph-mon[30312]: osdmap e344: 8 total, 8 up, 8 in 2024-03-24T11:46:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:48 smithi119 ceph-mon[33714]: pgmap v480: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:46:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:48 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:48 smithi119 ceph-mon[33714]: osdmap e345: 8 total, 8 up, 8 in 2024-03-24T11:46:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[42466]: pgmap v480: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:46:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[42466]: osdmap e345: 8 total, 8 up, 8 in 2024-03-24T11:46:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:48.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[30312]: pgmap v480: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:46:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[30312]: osdmap e345: 8 total, 8 up, 8 in 2024-03-24T11:46:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:50 smithi119 ceph-mon[33714]: pgmap v483: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:46:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:50 smithi119 ceph-mon[33714]: osdmap e346: 8 total, 8 up, 8 in 2024-03-24T11:46:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:50.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:50 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[30312]: pgmap v483: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:46:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[30312]: osdmap e346: 8 total, 8 up, 8 in 2024-03-24T11:46:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[42466]: pgmap v483: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:46:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[42466]: osdmap e346: 8 total, 8 up, 8 in 2024-03-24T11:46:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:50 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:51 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:51 smithi119 ceph-mon[33714]: osdmap e347: 8 total, 8 up, 8 in 2024-03-24T11:46:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:51 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[30312]: osdmap e347: 8 total, 8 up, 8 in 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[42466]: osdmap e347: 8 total, 8 up, 8 in 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2724607480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:51 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]: dispatch 2024-03-24T11:46:52.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:46:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:46:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:52 smithi119 ceph-mon[33714]: pgmap v486: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:52 smithi119 ceph-mon[33714]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:52 smithi119 ceph-mon[33714]: osdmap e348: 8 total, 8 up, 8 in 2024-03-24T11:46:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:52.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:52.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi114-82433-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[42466]: pgmap v486: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[42466]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[42466]: osdmap e348: 8 total, 8 up, 8 in 2024-03-24T11:46:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:52.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi114-82433-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:52.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[30312]: pgmap v486: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:46:52.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[30312]: from='client.56027 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi114-82433-63"}]': finished 2024-03-24T11:46:52.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[30312]: osdmap e348: 8 total, 8 up, 8 in 2024-03-24T11:46:52.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:52.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:52.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi114-82433-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi114-82433-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:53 smithi119 ceph-mon[33714]: osdmap e349: 8 total, 8 up, 8 in 2024-03-24T11:46:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi114-82433-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi114-82433-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[30312]: osdmap e349: 8 total, 8 up, 8 in 2024-03-24T11:46:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi114-82433-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:53.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:53.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi114-82433-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:46:53.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[42466]: osdmap e349: 8 total, 8 up, 8 in 2024-03-24T11:46:53.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi114-82433-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:54 smithi119 ceph-mon[33714]: pgmap v489: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:54 smithi119 ceph-mon[33714]: osdmap e350: 8 total, 8 up, 8 in 2024-03-24T11:46:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:54 smithi114 ceph-mon[30312]: pgmap v489: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:54 smithi114 ceph-mon[30312]: osdmap e350: 8 total, 8 up, 8 in 2024-03-24T11:46:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:54 smithi114 ceph-mon[42466]: pgmap v489: 356 pgs: 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:54 smithi114 ceph-mon[42466]: osdmap e350: 8 total, 8 up, 8 in 2024-03-24T11:46:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:46:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:46:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:46:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi114-82433-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:55.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:55 smithi114 ceph-mon[30312]: osdmap e351: 8 total, 8 up, 8 in 2024-03-24T11:46:55.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi114-82433-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:55.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:55 smithi114 ceph-mon[42466]: osdmap e351: 8 total, 8 up, 8 in 2024-03-24T11:46:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi114-82433-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:55 smithi119 ceph-mon[33714]: osdmap e351: 8 total, 8 up, 8 in 2024-03-24T11:46:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:56 smithi114 ceph-mon[30312]: pgmap v492: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:56 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:56 smithi114 ceph-mon[30312]: osdmap e352: 8 total, 8 up, 8 in 2024-03-24T11:46:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:56 smithi114 ceph-mon[42466]: pgmap v492: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:56 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:56 smithi114 ceph-mon[42466]: osdmap e352: 8 total, 8 up, 8 in 2024-03-24T11:46:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:56 smithi119 ceph-mon[33714]: pgmap v492: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:56 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:46:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:56 smithi119 ceph-mon[33714]: osdmap e352: 8 total, 8 up, 8 in 2024-03-24T11:46:57.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[30312]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T11:46:57.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:57.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:57.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:57.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[42466]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T11:46:57.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:57.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:57.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:57 smithi119 ceph-mon[33714]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T11:46:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:57.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[30312]: pgmap v495: 356 pgs: 356 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:46:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[30312]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T11:46:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:58.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[42466]: pgmap v495: 356 pgs: 356 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:58.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:58.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:46:58.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[42466]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T11:46:58.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:58.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:58.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:58 smithi119 ceph-mon[33714]: pgmap v495: 356 pgs: 356 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:46:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:58 smithi119 ceph-mon[33714]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T11:46:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]: dispatch 2024-03-24T11:46:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:58.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:59.098 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP163:head 2024-03-24T11:46:59.098 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:5d165639:::164:head 2024-03-24T11:46:59.098 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f43765fc:::165:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b4c720e9:::166:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:e694b040:::167:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:afa38db2:::168:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:77ba9f53:::169:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:87495034:::170:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:7c96bf0e:::171:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:dbe346cc:::172:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:e943ec24:::173:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f97a9c0c:::174:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:6f26e74d:::175:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:4f95e106:::176:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:0e6f2f8f:::177:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:05db05f1:::178:head 2024-03-24T11:46:59.099 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:38a78d66:::179:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:d095610b:::180:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:a1a9d709:::181:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:1e5d39db:::182:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:f7df4fb9:::183:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:03a7f161:::184:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:ba70721e:::185:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:28e5662d:::186:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:973d52de:::187:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:4303eb1c:::188:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:b990b48e:::189:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:29b8165b:::190:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:3547f197:::191:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:7e260936:::192:head 2024-03-24T11:46:59.100 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:1abec7b1:::193:head 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:10fdda93:::194:head 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:15817eea:::195:head 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:770bab57:::196:head 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:ed9e13e7:::197:head 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:71471a8f:::198:head 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: checking for 246:10fb1d02:::199:head 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8028 ms) 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,0 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: first is 1711280774 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,0 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,0 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,0 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,0 2024-03-24T11:46:59.101 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,1711280779,1711280780,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,1711280779,1711280780,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,1711280779,1711280780,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,1711280779,1711280780,1711280782,1711280783,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,1711280779,1711280780,1711280782,1711280783,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280774,1711280776,1711280777,1711280779,1711280780,1711280782,1711280783,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711280777,1711280779,1711280780,1711280782,1711280783,1711280785,1711280786,0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: first now 1711280777, trimmed 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20089 ms) 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: foo0 2024-03-24T11:46:59.102 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-24T11:46:59.103 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14053 ms) 2024-03-24T11:46:59.103 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-24T11:46:59.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:46:59.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:59.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]': finished 2024-03-24T11:46:59.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi114-82433-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]': finished 2024-03-24T11:46:59.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T11:46:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:46:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:46:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi114-82433-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:46:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:46:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/488232857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi114-82433-64"}]': finished 2024-03-24T11:46:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]': finished 2024-03-24T11:46:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T11:46:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:46:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:46:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi114-82433-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:46:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:46:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:46:59.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:46:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-49"}]: dispatch 2024-03-24T11:47:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[30312]: pgmap v498: 356 pgs: 356 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi114-82433-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:47:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[30312]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T11:47:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi114-82433-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:00.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[42466]: pgmap v498: 356 pgs: 356 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi114-82433-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:47:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[42466]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T11:47:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi114-82433-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:00 smithi119 ceph-mon[33714]: pgmap v498: 356 pgs: 356 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi114-82433-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:47:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:00 smithi119 ceph-mon[33714]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T11:47:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi114-82433-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T11:47:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi114-82433-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi114-82433-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:01.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T11:47:01.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:47:01.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:47:01.773 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T11:47:01.773 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:01.773 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:01.773 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi114-82433-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:01.774 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:01.774 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T11:47:01.774 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:47:01.774 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:47:02.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:47:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:47:02.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[42466]: pgmap v501: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[42466]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[30312]: pgmap v501: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[30312]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:02 smithi119 ceph-mon[33714]: pgmap v501: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:02 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:47:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:02 smithi119 ceph-mon[33714]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T11:47:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:02.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]': finished 2024-03-24T11:47:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T11:47:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]: dispatch 2024-03-24T11:47:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]: dispatch 2024-03-24T11:47:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]': finished 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]: dispatch 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]: dispatch 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-51"}]': finished 2024-03-24T11:47:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T11:47:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]: dispatch 2024-03-24T11:47:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]: dispatch 2024-03-24T11:47:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: pgmap v504: 364 pgs: 8 creating+peering, 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:47:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:47:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]': finished 2024-03-24T11:47:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T11:47:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: pgmap v504: 364 pgs: 8 creating+peering, 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]': finished 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:47:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:47:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: pgmap v504: 364 pgs: 8 creating+peering, 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:47:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:47:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-51", "mode": "writeback"}]': finished 2024-03-24T11:47:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T11:47:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]: dispatch 2024-03-24T11:47:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:47:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:47:04.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:47:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:47:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:47:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: pgmap v507: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:47:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:47:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: pgmap v507: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:47:06.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:47:06.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:06 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:47:06.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: pgmap v507: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2174462401' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi114-82433-65"}]': finished 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:47:06.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:06 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:47:07.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:07 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:47:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:47:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi114-82433-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:47:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T11:47:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:47:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:07.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:47:07.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:07.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:47:07.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:07 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:47:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: pgmap v510: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:47:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:47:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T11:47:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: pgmap v510: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:47:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:08.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:08.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:47:08.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:47:08.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T11:47:08.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:47:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: pgmap v510: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:47:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:47:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T11:47:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:47:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi114-82433-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:47:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:47:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T11:47:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:47:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:10 smithi114 ceph-mon[30312]: pgmap v513: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:47:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:10 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]': finished 2024-03-24T11:47:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:10 smithi114 ceph-mon[30312]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T11:47:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:10 smithi114 ceph-mon[42466]: pgmap v513: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:47:10.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:10 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]': finished 2024-03-24T11:47:10.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:10 smithi114 ceph-mon[42466]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T11:47:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:10 smithi119 ceph-mon[33714]: pgmap v513: 364 pgs: 8 unknown, 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:47:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:10 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-51","var": "target_max_objects","val": "1"}]': finished 2024-03-24T11:47:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:10 smithi119 ceph-mon[33714]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T11:47:11.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:11 smithi114 ceph-mon[30312]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T11:47:11.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:11.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:11 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:11 smithi114 ceph-mon[42466]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T11:47:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:11 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:11 smithi119 ceph-mon[33714]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T11:47:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:11 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:12.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:47:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ 2 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3009 ms) 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7010 ms) 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7055 ms) 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7037 ms) 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7045 ms) 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-24T11:47:12.148 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7018 ms) 2024-03-24T11:47:12.149 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-24T11:47:12.149 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7034 ms) 2024-03-24T11:47:12.149 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139097 ms total) 2024-03-24T11:47:12.149 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: 2024-03-24T11:47:12.149 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-24T11:47:12.149 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (304306 ms total) 2024-03-24T11:47:12.149 INFO:tasks.workunit.client.0.smithi114.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-24T11:47:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[30312]: pgmap v516: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-24T11:47:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[30312]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T11:47:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[42466]: pgmap v516: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-24T11:47:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[42466]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T11:47:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:12.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:12 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:12 smithi119 ceph-mon[33714]: pgmap v516: 356 pgs: 356 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:12 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-24T11:47:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:12 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:12 smithi119 ceph-mon[33714]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T11:47:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1525208602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:12.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:12 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]: dispatch 2024-03-24T11:47:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:13 smithi114 ceph-mon[30312]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:13 smithi114 ceph-mon[30312]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T11:47:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:13 smithi114 ceph-mon[42466]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:13 smithi114 ceph-mon[42466]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T11:47:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:13 smithi119 ceph-mon[33714]: from='client.56045 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi114-82433-66"}]': finished 2024-03-24T11:47:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:13 smithi119 ceph-mon[33714]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T11:47:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:14 smithi114 ceph-mon[30312]: pgmap v519: 356 pgs: 356 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:14 smithi114 ceph-mon[42466]: pgmap v519: 356 pgs: 356 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:14 smithi119 ceph-mon[33714]: pgmap v519: 356 pgs: 356 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:47:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:47:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:47:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:16 smithi114 ceph-mon[30312]: pgmap v520: 356 pgs: 356 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:16 smithi114 ceph-mon[42466]: pgmap v520: 356 pgs: 356 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:16 smithi119 ceph-mon[33714]: pgmap v520: 356 pgs: 356 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:18 smithi114 ceph-mon[30312]: pgmap v521: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:18 smithi114 ceph-mon[42466]: pgmap v521: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:18 smithi119 ceph-mon[33714]: pgmap v521: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:19 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:19.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:19 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[30312]: pgmap v522: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:47:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:20.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[30312]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T11:47:20.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[42466]: pgmap v522: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:47:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:20.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[42466]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T11:47:20.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:20 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:20 smithi119 ceph-mon[33714]: pgmap v522: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:20 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:47:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:20 smithi119 ceph-mon[33714]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T11:47:20.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:20 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]': finished 2024-03-24T11:47:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[42466]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T11:47:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:21.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]': finished 2024-03-24T11:47:21.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[30312]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T11:47:21.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:21.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:21.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:21.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]': finished 2024-03-24T11:47:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:21 smithi119 ceph-mon[33714]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T11:47:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-51"}]: dispatch 2024-03-24T11:47:22.057 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:47:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:47:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:22 smithi114 ceph-mon[30312]: pgmap v525: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:22 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-24T11:47:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:22 smithi114 ceph-mon[30312]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T11:47:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:22 smithi114 ceph-mon[42466]: pgmap v525: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:22 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-24T11:47:22.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:22 smithi114 ceph-mon[42466]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T11:47:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:22 smithi119 ceph-mon[33714]: pgmap v525: 356 pgs: 356 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:22 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-24T11:47:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:22 smithi119 ceph-mon[33714]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T11:47:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:23 smithi119 ceph-mon[33714]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T11:47:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:23.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:23 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[30312]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T11:47:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[42466]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T11:47:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:23.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:23 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: pgmap v528: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:47:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T11:47:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-53"}]: dispatch 2024-03-24T11:47:24.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-53"}]: dispatch 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: pgmap v528: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-53"}]: dispatch 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-53"}]: dispatch 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: pgmap v528: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:47:24.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T11:47:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-53"}]: dispatch 2024-03-24T11:47:24.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-53"}]: dispatch 2024-03-24T11:47:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:47:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:47:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:47:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:25 smithi119 ceph-mon[33714]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T11:47:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:25 smithi114 ceph-mon[30312]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T11:47:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:25 smithi114 ceph-mon[42466]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T11:47:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:26 smithi119 ceph-mon[33714]: pgmap v531: 324 pgs: 324 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:47:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:26 smithi119 ceph-mon[33714]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T11:47:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[30312]: pgmap v531: 324 pgs: 324 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:47:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[30312]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T11:47:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:26.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[42466]: pgmap v531: 324 pgs: 324 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:47:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[42466]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T11:47:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:26.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:27 smithi119 ceph-mon[33714]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T11:47:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-55"}]: dispatch 2024-03-24T11:47:27.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-55"}]: dispatch 2024-03-24T11:47:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[30312]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T11:47:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-55"}]: dispatch 2024-03-24T11:47:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-55"}]: dispatch 2024-03-24T11:47:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[42466]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T11:47:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-55"}]: dispatch 2024-03-24T11:47:27.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-55"}]: dispatch 2024-03-24T11:47:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:28 smithi119 ceph-mon[33714]: pgmap v534: 356 pgs: 356 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-24T11:47:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:28 smithi119 ceph-mon[33714]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T11:47:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:28 smithi114 ceph-mon[30312]: pgmap v534: 356 pgs: 356 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-24T11:47:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:28 smithi114 ceph-mon[30312]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T11:47:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:28 smithi114 ceph-mon[42466]: pgmap v534: 356 pgs: 356 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-24T11:47:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:28 smithi114 ceph-mon[42466]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T11:47:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:29 smithi119 ceph-mon[33714]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T11:47:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:29 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:29 smithi114 ceph-mon[30312]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T11:47:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:29 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:29 smithi114 ceph-mon[42466]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T11:47:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:29.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:29 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: pgmap v537: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:47:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:30.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:30.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T11:47:30.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:30.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:30.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-57"}]: dispatch 2024-03-24T11:47:30.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-57"}]: dispatch 2024-03-24T11:47:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: pgmap v537: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:47:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T11:47:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:30.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-57"}]: dispatch 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-57"}]: dispatch 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: pgmap v537: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-57"}]: dispatch 2024-03-24T11:47:30.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-57"}]: dispatch 2024-03-24T11:47:31.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:31 smithi114 ceph-mon[30312]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T11:47:31.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:31 smithi114 ceph-mon[42466]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T11:47:31.800 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:31 smithi119 ceph-mon[33714]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T11:47:32.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:47:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:47:32.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[30312]: pgmap v540: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:32.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[30312]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T11:47:32.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:32.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:32.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[42466]: pgmap v540: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:32.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[42466]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T11:47:32.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:32.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:32 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:32 smithi119 ceph-mon[33714]: pgmap v540: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:32 smithi119 ceph-mon[33714]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T11:47:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:32.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:32 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T11:47:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:33.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-59"}]: dispatch 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-59"}]: dispatch 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:33.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-59"}]: dispatch 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-59"}]: dispatch 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:33.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-59"}]: dispatch 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-59"}]: dispatch 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[42466]: pgmap v543: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[42466]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[30312]: pgmap v543: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[30312]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:34.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:34 smithi119 ceph-mon[33714]: pgmap v543: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:47:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:34 smithi119 ceph-mon[33714]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T11:47:34.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:34.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:47:34.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:47:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:47:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:47:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:35 smithi114 ceph-mon[30312]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T11:47:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:35.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:35.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:35 smithi114 ceph-mon[42466]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T11:47:35.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:35.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:35 smithi119 ceph-mon[33714]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T11:47:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:35.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[30312]: pgmap v546: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:47:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[30312]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T11:47:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[42466]: pgmap v546: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:47:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[42466]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T11:47:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:36.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:36.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:36 smithi119 ceph-mon[33714]: pgmap v546: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:47:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:36 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:36 smithi119 ceph-mon[33714]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T11:47:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:36.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[42466]: osdmap e387: 8 total, 8 up, 8 in 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-61"}]: dispatch 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-61"}]: dispatch 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[30312]: osdmap e387: 8 total, 8 up, 8 in 2024-03-24T11:47:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:37.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:37.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-61"}]: dispatch 2024-03-24T11:47:37.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-61"}]: dispatch 2024-03-24T11:47:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:47:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:37 smithi119 ceph-mon[33714]: osdmap e387: 8 total, 8 up, 8 in 2024-03-24T11:47:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:47:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-61"}]: dispatch 2024-03-24T11:47:37.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-61"}]: dispatch 2024-03-24T11:47:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:38 smithi114 ceph-mon[30312]: pgmap v549: 356 pgs: 356 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T11:47:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:38 smithi114 ceph-mon[30312]: osdmap e388: 8 total, 8 up, 8 in 2024-03-24T11:47:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:38 smithi114 ceph-mon[42466]: pgmap v549: 356 pgs: 356 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T11:47:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:38 smithi114 ceph-mon[42466]: osdmap e388: 8 total, 8 up, 8 in 2024-03-24T11:47:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:38 smithi119 ceph-mon[33714]: pgmap v549: 356 pgs: 356 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T11:47:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:38 smithi119 ceph-mon[33714]: osdmap e388: 8 total, 8 up, 8 in 2024-03-24T11:47:39.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:39 smithi114 ceph-mon[30312]: osdmap e389: 8 total, 8 up, 8 in 2024-03-24T11:47:39.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:39.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:39.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:39 smithi114 ceph-mon[42466]: osdmap e389: 8 total, 8 up, 8 in 2024-03-24T11:47:39.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:39.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:39 smithi119 ceph-mon[33714]: osdmap e389: 8 total, 8 up, 8 in 2024-03-24T11:47:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:39.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:47:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[42466]: pgmap v552: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:47:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[42466]: osdmap e390: 8 total, 8 up, 8 in 2024-03-24T11:47:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:40.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[30312]: pgmap v552: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:47:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[30312]: osdmap e390: 8 total, 8 up, 8 in 2024-03-24T11:47:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:40.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:40 smithi119 ceph-mon[33714]: pgmap v552: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:47:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:47:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:40 smithi119 ceph-mon[33714]: osdmap e390: 8 total, 8 up, 8 in 2024-03-24T11:47:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:47:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:40.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:47:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:47:41.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:41 smithi114 ceph-mon[30312]: osdmap e391: 8 total, 8 up, 8 in 2024-03-24T11:47:41.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:47:41.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:41 smithi114 ceph-mon[42466]: osdmap e391: 8 total, 8 up, 8 in 2024-03-24T11:47:41.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:47:41.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:41 smithi119 ceph-mon[33714]: osdmap e391: 8 total, 8 up, 8 in 2024-03-24T11:47:42.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:47:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:47:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:42 smithi114 ceph-mon[30312]: pgmap v555: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:42 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:42 smithi114 ceph-mon[30312]: osdmap e392: 8 total, 8 up, 8 in 2024-03-24T11:47:42.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:42 smithi114 ceph-mon[42466]: pgmap v555: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:42.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:42 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:42.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:42 smithi114 ceph-mon[42466]: osdmap e392: 8 total, 8 up, 8 in 2024-03-24T11:47:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:42 smithi119 ceph-mon[33714]: pgmap v555: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:47:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:42 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:47:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:42 smithi119 ceph-mon[33714]: osdmap e392: 8 total, 8 up, 8 in 2024-03-24T11:47:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:43.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:43 smithi114 ceph-mon[30312]: osdmap e393: 8 total, 8 up, 8 in 2024-03-24T11:47:43.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:43.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:43 smithi114 ceph-mon[42466]: osdmap e393: 8 total, 8 up, 8 in 2024-03-24T11:47:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:43 smithi119 ceph-mon[33714]: osdmap e393: 8 total, 8 up, 8 in 2024-03-24T11:47:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:44 smithi114 ceph-mon[30312]: pgmap v558: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-24T11:47:44.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:44 smithi114 ceph-mon[30312]: osdmap e394: 8 total, 8 up, 8 in 2024-03-24T11:47:44.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:47:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:47:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:47:44.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:44 smithi114 ceph-mon[42466]: pgmap v558: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-24T11:47:44.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:44 smithi114 ceph-mon[42466]: osdmap e394: 8 total, 8 up, 8 in 2024-03-24T11:47:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:44 smithi119 ceph-mon[33714]: pgmap v558: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-24T11:47:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:44 smithi119 ceph-mon[33714]: osdmap e394: 8 total, 8 up, 8 in 2024-03-24T11:47:45.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:45 smithi114 ceph-mon[42466]: osdmap e395: 8 total, 8 up, 8 in 2024-03-24T11:47:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:45 smithi114 ceph-mon[30312]: osdmap e395: 8 total, 8 up, 8 in 2024-03-24T11:47:45.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:45 smithi119 ceph-mon[33714]: osdmap e395: 8 total, 8 up, 8 in 2024-03-24T11:47:46.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:46 smithi114 ceph-mon[30312]: pgmap v561: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-24T11:47:46.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:46 smithi114 ceph-mon[42466]: pgmap v561: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-24T11:47:46.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:46 smithi119 ceph-mon[33714]: pgmap v561: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-24T11:47:48.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:48 smithi114 ceph-mon[30312]: pgmap v562: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-24T11:47:48.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:48 smithi114 ceph-mon[30312]: osdmap e396: 8 total, 8 up, 8 in 2024-03-24T11:47:48.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:48 smithi114 ceph-mon[42466]: pgmap v562: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-24T11:47:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:48 smithi114 ceph-mon[42466]: osdmap e396: 8 total, 8 up, 8 in 2024-03-24T11:47:48.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:48.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:48 smithi119 ceph-mon[33714]: pgmap v562: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-24T11:47:48.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:48 smithi119 ceph-mon[33714]: osdmap e396: 8 total, 8 up, 8 in 2024-03-24T11:47:48.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:50 smithi114 ceph-mon[30312]: pgmap v564: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-24T11:47:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:50 smithi114 ceph-mon[42466]: pgmap v564: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-24T11:47:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:50 smithi119 ceph-mon[33714]: pgmap v564: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-24T11:47:52.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:47:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:47:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:52 smithi114 ceph-mon[30312]: pgmap v565: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 552 B/s wr, 2 op/s 2024-03-24T11:47:52.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:52 smithi114 ceph-mon[42466]: pgmap v565: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 552 B/s wr, 2 op/s 2024-03-24T11:47:52.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:52 smithi119 ceph-mon[33714]: pgmap v565: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 552 B/s wr, 2 op/s 2024-03-24T11:47:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:54.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:54 smithi114 ceph-mon[30312]: pgmap v566: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 486 B/s wr, 2 op/s 2024-03-24T11:47:54.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:54 smithi114 ceph-mon[42466]: pgmap v566: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 486 B/s wr, 2 op/s 2024-03-24T11:47:54.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:47:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:47:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:47:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:54 smithi119 ceph-mon[33714]: pgmap v566: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 486 B/s wr, 2 op/s 2024-03-24T11:47:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:55 smithi114 ceph-mon[30312]: osdmap e397: 8 total, 8 up, 8 in 2024-03-24T11:47:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:55 smithi114 ceph-mon[42466]: osdmap e397: 8 total, 8 up, 8 in 2024-03-24T11:47:55.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:55 smithi119 ceph-mon[33714]: osdmap e397: 8 total, 8 up, 8 in 2024-03-24T11:47:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:56 smithi114 ceph-mon[30312]: pgmap v568: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:56 smithi114 ceph-mon[42466]: pgmap v568: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:56.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:56 smithi119 ceph-mon[33714]: pgmap v568: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:58 smithi114 ceph-mon[30312]: pgmap v569: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:47:58 smithi114 ceph-mon[30312]: osdmap e398: 8 total, 8 up, 8 in 2024-03-24T11:47:58.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:58 smithi114 ceph-mon[42466]: pgmap v569: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:58.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:47:58 smithi114 ceph-mon[42466]: osdmap e398: 8 total, 8 up, 8 in 2024-03-24T11:47:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:58 smithi119 ceph-mon[33714]: pgmap v569: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:47:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:47:58 smithi119 ceph-mon[33714]: osdmap e398: 8 total, 8 up, 8 in 2024-03-24T11:48:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:00 smithi114 ceph-mon[30312]: pgmap v571: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:00.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:00 smithi114 ceph-mon[42466]: pgmap v571: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:00 smithi119 ceph-mon[33714]: pgmap v571: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:02.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:48:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:48:02.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:02 smithi114 ceph-mon[30312]: pgmap v572: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:02.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:02 smithi114 ceph-mon[42466]: pgmap v572: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:02 smithi119 ceph-mon[33714]: pgmap v572: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:04.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:04 smithi114 ceph-mon[30312]: pgmap v573: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:04.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:04 smithi114 ceph-mon[42466]: pgmap v573: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:04.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:48:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:48:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:48:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:04 smithi119 ceph-mon[33714]: pgmap v573: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:05 smithi114 ceph-mon[30312]: osdmap e399: 8 total, 8 up, 8 in 2024-03-24T11:48:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:05 smithi114 ceph-mon[42466]: osdmap e399: 8 total, 8 up, 8 in 2024-03-24T11:48:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:05 smithi119 ceph-mon[33714]: osdmap e399: 8 total, 8 up, 8 in 2024-03-24T11:48:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:06 smithi114 ceph-mon[30312]: pgmap v575: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:48:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:06 smithi114 ceph-mon[42466]: pgmap v575: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:48:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:06 smithi119 ceph-mon[33714]: pgmap v575: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:48:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:08 smithi114 ceph-mon[30312]: pgmap v576: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:08 smithi114 ceph-mon[30312]: osdmap e400: 8 total, 8 up, 8 in 2024-03-24T11:48:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:08 smithi114 ceph-mon[42466]: pgmap v576: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:08 smithi114 ceph-mon[42466]: osdmap e400: 8 total, 8 up, 8 in 2024-03-24T11:48:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:08 smithi119 ceph-mon[33714]: pgmap v576: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:08 smithi119 ceph-mon[33714]: osdmap e400: 8 total, 8 up, 8 in 2024-03-24T11:48:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:10 smithi114 ceph-mon[30312]: pgmap v578: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:48:10.757 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:10 smithi114 ceph-mon[42466]: pgmap v578: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:48:10.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:10 smithi119 ceph-mon[33714]: pgmap v578: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:48:12.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:48:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:48:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:12 smithi114 ceph-mon[30312]: pgmap v579: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:48:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:12 smithi114 ceph-mon[42466]: pgmap v579: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:48:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:12 smithi119 ceph-mon[33714]: pgmap v579: 356 pgs: 356 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:48:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:13.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:13.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:14 smithi119 ceph-mon[33714]: pgmap v580: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:48:14.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:14 smithi114 ceph-mon[30312]: pgmap v580: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:48:14.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:14 smithi114 ceph-mon[42466]: pgmap v580: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:48:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:48:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:48:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:48:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-63"}]: dispatch 2024-03-24T11:48:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-63"}]: dispatch 2024-03-24T11:48:15.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:15.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:15.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-63"}]: dispatch 2024-03-24T11:48:15.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-63"}]: dispatch 2024-03-24T11:48:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-63"}]: dispatch 2024-03-24T11:48:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-63"}]: dispatch 2024-03-24T11:48:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:16 smithi119 ceph-mon[33714]: pgmap v581: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:16 smithi119 ceph-mon[33714]: osdmap e401: 8 total, 8 up, 8 in 2024-03-24T11:48:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:16 smithi114 ceph-mon[30312]: pgmap v581: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:16.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:16 smithi114 ceph-mon[30312]: osdmap e401: 8 total, 8 up, 8 in 2024-03-24T11:48:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:16 smithi114 ceph-mon[42466]: pgmap v581: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:16.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:16 smithi114 ceph-mon[42466]: osdmap e401: 8 total, 8 up, 8 in 2024-03-24T11:48:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:17 smithi119 ceph-mon[33714]: osdmap e402: 8 total, 8 up, 8 in 2024-03-24T11:48:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:17 smithi119 ceph-mon[33714]: osdmap e403: 8 total, 8 up, 8 in 2024-03-24T11:48:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[30312]: osdmap e402: 8 total, 8 up, 8 in 2024-03-24T11:48:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:17.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:17.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[30312]: osdmap e403: 8 total, 8 up, 8 in 2024-03-24T11:48:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[42466]: osdmap e402: 8 total, 8 up, 8 in 2024-03-24T11:48:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:17.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:17 smithi114 ceph-mon[42466]: osdmap e403: 8 total, 8 up, 8 in 2024-03-24T11:48:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: pgmap v584: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:48:18.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:18 smithi119 ceph-mon[33714]: osdmap e404: 8 total, 8 up, 8 in 2024-03-24T11:48:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: pgmap v584: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[30312]: osdmap e404: 8 total, 8 up, 8 in 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: pgmap v584: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:18.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:48:18.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:18 smithi114 ceph-mon[42466]: osdmap e404: 8 total, 8 up, 8 in 2024-03-24T11:48:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:19 smithi119 ceph-mon[33714]: pgmap v587: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:19.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:19 smithi114 ceph-mon[30312]: pgmap v587: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:19.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:19 smithi114 ceph-mon[42466]: pgmap v587: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:20 smithi119 ceph-mon[33714]: osdmap e405: 8 total, 8 up, 8 in 2024-03-24T11:48:20.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:20 smithi114 ceph-mon[30312]: osdmap e405: 8 total, 8 up, 8 in 2024-03-24T11:48:20.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:20 smithi114 ceph-mon[42466]: osdmap e405: 8 total, 8 up, 8 in 2024-03-24T11:48:21.837 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:21 smithi119 ceph-mon[33714]: osdmap e406: 8 total, 8 up, 8 in 2024-03-24T11:48:21.837 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:21 smithi119 ceph-mon[33714]: pgmap v590: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:21 smithi114 ceph-mon[30312]: osdmap e406: 8 total, 8 up, 8 in 2024-03-24T11:48:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:21 smithi114 ceph-mon[30312]: pgmap v590: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:21.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:21 smithi114 ceph-mon[42466]: osdmap e406: 8 total, 8 up, 8 in 2024-03-24T11:48:21.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:21 smithi114 ceph-mon[42466]: pgmap v590: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:22.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:48:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:48:22.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:22 smithi114 ceph-mon[30312]: osdmap e407: 8 total, 8 up, 8 in 2024-03-24T11:48:22.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:22 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:22 smithi114 ceph-mon[42466]: osdmap e407: 8 total, 8 up, 8 in 2024-03-24T11:48:22.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:22 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:23.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:22 smithi119 ceph-mon[33714]: osdmap e407: 8 total, 8 up, 8 in 2024-03-24T11:48:23.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:22 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:23 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:23.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:23 smithi114 ceph-mon[30312]: pgmap v592: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-24T11:48:23.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:23 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:23.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:23 smithi114 ceph-mon[42466]: pgmap v592: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-24T11:48:24.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:23 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:24.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:23 smithi119 ceph-mon[33714]: pgmap v592: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-24T11:48:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:48:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:48:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:48:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:25 smithi114 ceph-mon[30312]: pgmap v593: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T11:48:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:25 smithi114 ceph-mon[42466]: pgmap v593: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T11:48:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:25 smithi119 ceph-mon[33714]: pgmap v593: 356 pgs: 356 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-24T11:48:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:27 smithi114 ceph-mon[30312]: pgmap v594: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T11:48:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:27 smithi114 ceph-mon[30312]: osdmap e408: 8 total, 8 up, 8 in 2024-03-24T11:48:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:27 smithi114 ceph-mon[42466]: pgmap v594: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T11:48:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:27 smithi114 ceph-mon[42466]: osdmap e408: 8 total, 8 up, 8 in 2024-03-24T11:48:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:27 smithi119 ceph-mon[33714]: pgmap v594: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T11:48:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:27 smithi119 ceph-mon[33714]: osdmap e408: 8 total, 8 up, 8 in 2024-03-24T11:48:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:29 smithi114 ceph-mon[30312]: pgmap v596: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-24T11:48:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:29 smithi114 ceph-mon[42466]: pgmap v596: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-24T11:48:30.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:29 smithi119 ceph-mon[33714]: pgmap v596: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-24T11:48:32.107 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:48:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:48:32.107 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:31 smithi119 ceph-mon[33714]: pgmap v597: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:32.107 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:32.107 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:31 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:32.107 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-65"}]: dispatch 2024-03-24T11:48:32.107 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:31 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-65"}]: dispatch 2024-03-24T11:48:32.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[30312]: pgmap v597: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:32.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:32.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:32.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-65"}]: dispatch 2024-03-24T11:48:32.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-65"}]: dispatch 2024-03-24T11:48:32.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[42466]: pgmap v597: 356 pgs: 356 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:48:32.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:32.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:32.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-65"}]: dispatch 2024-03-24T11:48:32.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:31 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-65"}]: dispatch 2024-03-24T11:48:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:32 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:32 smithi114 ceph-mon[30312]: osdmap e409: 8 total, 8 up, 8 in 2024-03-24T11:48:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:32 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:32 smithi114 ceph-mon[42466]: osdmap e409: 8 total, 8 up, 8 in 2024-03-24T11:48:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:32 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:32 smithi119 ceph-mon[33714]: osdmap e409: 8 total, 8 up, 8 in 2024-03-24T11:48:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[30312]: pgmap v599: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[30312]: osdmap e410: 8 total, 8 up, 8 in 2024-03-24T11:48:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:34.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[42466]: pgmap v599: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[42466]: osdmap e410: 8 total, 8 up, 8 in 2024-03-24T11:48:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:33 smithi119 ceph-mon[33714]: pgmap v599: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:33 smithi119 ceph-mon[33714]: osdmap e410: 8 total, 8 up, 8 in 2024-03-24T11:48:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:34.922 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:48:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:48:35.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:35.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[30312]: osdmap e411: 8 total, 8 up, 8 in 2024-03-24T11:48:35.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:35.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:35.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:48:35.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:35.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[42466]: osdmap e411: 8 total, 8 up, 8 in 2024-03-24T11:48:35.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:35.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:35.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:34 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:34 smithi119 ceph-mon[33714]: osdmap e411: 8 total, 8 up, 8 in 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:48:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[30312]: pgmap v602: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[30312]: osdmap e412: 8 total, 8 up, 8 in 2024-03-24T11:48:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-67"}]: dispatch 2024-03-24T11:48:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-67"}]: dispatch 2024-03-24T11:48:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[42466]: pgmap v602: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[42466]: osdmap e412: 8 total, 8 up, 8 in 2024-03-24T11:48:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-67"}]: dispatch 2024-03-24T11:48:36.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-67"}]: dispatch 2024-03-24T11:48:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:35 smithi119 ceph-mon[33714]: pgmap v602: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:35 smithi119 ceph-mon[33714]: osdmap e412: 8 total, 8 up, 8 in 2024-03-24T11:48:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-67"}]: dispatch 2024-03-24T11:48:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-67"}]: dispatch 2024-03-24T11:48:37.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:36 smithi114 ceph-mon[30312]: osdmap e413: 8 total, 8 up, 8 in 2024-03-24T11:48:37.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:36 smithi114 ceph-mon[42466]: osdmap e413: 8 total, 8 up, 8 in 2024-03-24T11:48:37.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:36 smithi119 ceph-mon[33714]: osdmap e413: 8 total, 8 up, 8 in 2024-03-24T11:48:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[42466]: pgmap v605: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:48:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[42466]: osdmap e414: 8 total, 8 up, 8 in 2024-03-24T11:48:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[30312]: pgmap v605: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:48:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[30312]: osdmap e414: 8 total, 8 up, 8 in 2024-03-24T11:48:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:38.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:37 smithi119 ceph-mon[33714]: pgmap v605: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:48:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:37 smithi119 ceph-mon[33714]: osdmap e414: 8 total, 8 up, 8 in 2024-03-24T11:48:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[42466]: osdmap e415: 8 total, 8 up, 8 in 2024-03-24T11:48:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-69"}]: dispatch 2024-03-24T11:48:39.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-69"}]: dispatch 2024-03-24T11:48:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[30312]: osdmap e415: 8 total, 8 up, 8 in 2024-03-24T11:48:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:39.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:39.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:39.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-69"}]: dispatch 2024-03-24T11:48:39.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-69"}]: dispatch 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:38 smithi119 ceph-mon[33714]: osdmap e415: 8 total, 8 up, 8 in 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:48:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-69"}]: dispatch 2024-03-24T11:48:39.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-69"}]: dispatch 2024-03-24T11:48:40.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:39 smithi114 ceph-mon[30312]: pgmap v608: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:48:40.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:39 smithi114 ceph-mon[30312]: osdmap e416: 8 total, 8 up, 8 in 2024-03-24T11:48:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:39 smithi114 ceph-mon[42466]: pgmap v608: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:48:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:39 smithi114 ceph-mon[42466]: osdmap e416: 8 total, 8 up, 8 in 2024-03-24T11:48:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:39 smithi119 ceph-mon[33714]: pgmap v608: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:48:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:39 smithi119 ceph-mon[33714]: osdmap e416: 8 total, 8 up, 8 in 2024-03-24T11:48:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:40 smithi119 ceph-mon[33714]: osdmap e417: 8 total, 8 up, 8 in 2024-03-24T11:48:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:41.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:40 smithi119 ceph-mon[33714]: osdmap e418: 8 total, 8 up, 8 in 2024-03-24T11:48:41.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:41.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[42466]: osdmap e417: 8 total, 8 up, 8 in 2024-03-24T11:48:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[42466]: osdmap e418: 8 total, 8 up, 8 in 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[30312]: osdmap e417: 8 total, 8 up, 8 in 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[30312]: osdmap e418: 8 total, 8 up, 8 in 2024-03-24T11:48:41.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:48:41.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:42.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:48:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:48:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:41 smithi119 ceph-mon[33714]: pgmap v611: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:42.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:41 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:42.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:48:42.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:41 smithi119 ceph-mon[33714]: osdmap e419: 8 total, 8 up, 8 in 2024-03-24T11:48:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:41 smithi114 ceph-mon[30312]: pgmap v611: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:41 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:48:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:41 smithi114 ceph-mon[30312]: osdmap e419: 8 total, 8 up, 8 in 2024-03-24T11:48:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:42 smithi114 ceph-mon[42466]: pgmap v611: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:48:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:42 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:48:42.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:42 smithi114 ceph-mon[42466]: osdmap e419: 8 total, 8 up, 8 in 2024-03-24T11:48:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:44 smithi119 ceph-mon[33714]: pgmap v614: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:48:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:44 smithi119 ceph-mon[33714]: osdmap e420: 8 total, 8 up, 8 in 2024-03-24T11:48:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:44 smithi114 ceph-mon[30312]: pgmap v614: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:48:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:44 smithi114 ceph-mon[30312]: osdmap e420: 8 total, 8 up, 8 in 2024-03-24T11:48:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:44 smithi114 ceph-mon[42466]: pgmap v614: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:48:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:44 smithi114 ceph-mon[42466]: osdmap e420: 8 total, 8 up, 8 in 2024-03-24T11:48:44.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:48:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:48:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:48:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:46 smithi119 ceph-mon[33714]: pgmap v616: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 630 B/s wr, 1 op/s 2024-03-24T11:48:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:46 smithi114 ceph-mon[30312]: pgmap v616: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 630 B/s wr, 1 op/s 2024-03-24T11:48:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:46 smithi114 ceph-mon[42466]: pgmap v616: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 630 B/s wr, 1 op/s 2024-03-24T11:48:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:48 smithi119 ceph-mon[33714]: pgmap v617: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-24T11:48:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:48 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:48 smithi114 ceph-mon[30312]: pgmap v617: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-24T11:48:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:48 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:48 smithi114 ceph-mon[42466]: pgmap v617: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-24T11:48:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:48 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:48:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:50 smithi119 ceph-mon[33714]: pgmap v618: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-24T11:48:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:50 smithi114 ceph-mon[30312]: pgmap v618: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-24T11:48:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:50 smithi114 ceph-mon[42466]: pgmap v618: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-24T11:48:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:52 smithi119 ceph-mon[33714]: pgmap v619: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 692 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:48:52.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:48:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:48:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:52 smithi114 ceph-mon[30312]: pgmap v619: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 692 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:48:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:52 smithi114 ceph-mon[42466]: pgmap v619: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 692 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-24T11:48:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:53.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:53.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:54 smithi119 ceph-mon[33714]: pgmap v620: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-24T11:48:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:54 smithi114 ceph-mon[30312]: pgmap v620: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-24T11:48:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:54 smithi114 ceph-mon[42466]: pgmap v620: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-24T11:48:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:48:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:48:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:48:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:56 smithi119 ceph-mon[33714]: pgmap v621: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-24T11:48:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:56 smithi114 ceph-mon[30312]: pgmap v621: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-24T11:48:56.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:56 smithi114 ceph-mon[42466]: pgmap v621: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-24T11:48:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:48:58 smithi119 ceph-mon[33714]: pgmap v622: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-24T11:48:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:48:58 smithi114 ceph-mon[30312]: pgmap v622: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-24T11:48:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:48:58 smithi114 ceph-mon[42466]: pgmap v622: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-24T11:49:00.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:00 smithi119 ceph-mon[33714]: pgmap v623: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:00.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:00 smithi114 ceph-mon[30312]: pgmap v623: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:00.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:00 smithi114 ceph-mon[42466]: pgmap v623: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:02.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:02 smithi119 ceph-mon[33714]: pgmap v624: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:02.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:49:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:49:02.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:02 smithi114 ceph-mon[30312]: pgmap v624: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:02.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:02 smithi114 ceph-mon[42466]: pgmap v624: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:03.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:03.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:03.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:04 smithi119 ceph-mon[33714]: pgmap v625: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-71"}]: dispatch 2024-03-24T11:49:04.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:04 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-71"}]: dispatch 2024-03-24T11:49:04.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[42466]: pgmap v625: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-71"}]: dispatch 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-71"}]: dispatch 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[30312]: pgmap v625: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-71"}]: dispatch 2024-03-24T11:49:04.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-71"}]: dispatch 2024-03-24T11:49:04.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:49:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:49:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:49:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[42466]: osdmap e421: 8 total, 8 up, 8 in 2024-03-24T11:49:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[42466]: osdmap e422: 8 total, 8 up, 8 in 2024-03-24T11:49:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[30312]: osdmap e421: 8 total, 8 up, 8 in 2024-03-24T11:49:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[30312]: osdmap e422: 8 total, 8 up, 8 in 2024-03-24T11:49:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:05.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:05 smithi119 ceph-mon[33714]: osdmap e421: 8 total, 8 up, 8 in 2024-03-24T11:49:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:05 smithi119 ceph-mon[33714]: osdmap e422: 8 total, 8 up, 8 in 2024-03-24T11:49:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[30312]: pgmap v627: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[30312]: osdmap e423: 8 total, 8 up, 8 in 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[42466]: pgmap v627: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[42466]: osdmap e423: 8 total, 8 up, 8 in 2024-03-24T11:49:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:06 smithi119 ceph-mon[33714]: pgmap v627: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-24T11:49:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:06 smithi119 ceph-mon[33714]: osdmap e423: 8 total, 8 up, 8 in 2024-03-24T11:49:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:08 smithi114 ceph-mon[30312]: pgmap v630: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:08 smithi114 ceph-mon[30312]: osdmap e424: 8 total, 8 up, 8 in 2024-03-24T11:49:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:08 smithi114 ceph-mon[42466]: pgmap v630: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:08 smithi114 ceph-mon[42466]: osdmap e424: 8 total, 8 up, 8 in 2024-03-24T11:49:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:08 smithi119 ceph-mon[33714]: pgmap v630: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:08.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:08 smithi119 ceph-mon[33714]: osdmap e424: 8 total, 8 up, 8 in 2024-03-24T11:49:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:09 smithi114 ceph-mon[30312]: osdmap e425: 8 total, 8 up, 8 in 2024-03-24T11:49:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:09 smithi114 ceph-mon[42466]: osdmap e425: 8 total, 8 up, 8 in 2024-03-24T11:49:09.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:09 smithi119 ceph-mon[33714]: osdmap e425: 8 total, 8 up, 8 in 2024-03-24T11:49:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:10 smithi114 ceph-mon[30312]: pgmap v633: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:49:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:10 smithi114 ceph-mon[42466]: pgmap v633: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:49:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:10 smithi119 ceph-mon[33714]: pgmap v633: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:49:12.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:12 smithi119 ceph-mon[33714]: pgmap v634: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 883 B/s wr, 1 op/s 2024-03-24T11:49:12.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:49:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:49:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:12 smithi114 ceph-mon[30312]: pgmap v634: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 883 B/s wr, 1 op/s 2024-03-24T11:49:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:12 smithi114 ceph-mon[42466]: pgmap v634: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 883 B/s wr, 1 op/s 2024-03-24T11:49:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:14 smithi114 ceph-mon[30312]: pgmap v635: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2024-03-24T11:49:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:14 smithi114 ceph-mon[42466]: pgmap v635: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2024-03-24T11:49:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:14 smithi119 ceph-mon[33714]: pgmap v635: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2024-03-24T11:49:14.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:49:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:49:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:49:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:16 smithi114 ceph-mon[30312]: pgmap v636: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:16 smithi114 ceph-mon[42466]: pgmap v636: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:16 smithi119 ceph-mon[33714]: pgmap v636: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:49:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:18 smithi114 ceph-mon[30312]: pgmap v637: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 209 B/s wr, 2 op/s 2024-03-24T11:49:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:18 smithi114 ceph-mon[42466]: pgmap v637: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 209 B/s wr, 2 op/s 2024-03-24T11:49:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:18 smithi119 ceph-mon[33714]: pgmap v637: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 209 B/s wr, 2 op/s 2024-03-24T11:49:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-73"}]: dispatch 2024-03-24T11:49:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-73"}]: dispatch 2024-03-24T11:49:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-73"}]: dispatch 2024-03-24T11:49:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-73"}]: dispatch 2024-03-24T11:49:19.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:19 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-73"}]: dispatch 2024-03-24T11:49:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:19 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-73"}]: dispatch 2024-03-24T11:49:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:20 smithi114 ceph-mon[30312]: pgmap v638: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 189 B/s wr, 2 op/s 2024-03-24T11:49:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:20 smithi114 ceph-mon[30312]: osdmap e426: 8 total, 8 up, 8 in 2024-03-24T11:49:20.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:20 smithi114 ceph-mon[42466]: pgmap v638: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 189 B/s wr, 2 op/s 2024-03-24T11:49:20.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:20 smithi114 ceph-mon[42466]: osdmap e426: 8 total, 8 up, 8 in 2024-03-24T11:49:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:20 smithi119 ceph-mon[33714]: pgmap v638: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 189 B/s wr, 2 op/s 2024-03-24T11:49:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:20 smithi119 ceph-mon[33714]: osdmap e426: 8 total, 8 up, 8 in 2024-03-24T11:49:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[30312]: osdmap e427: 8 total, 8 up, 8 in 2024-03-24T11:49:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[30312]: osdmap e428: 8 total, 8 up, 8 in 2024-03-24T11:49:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[42466]: osdmap e427: 8 total, 8 up, 8 in 2024-03-24T11:49:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:21.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:21 smithi114 ceph-mon[42466]: osdmap e428: 8 total, 8 up, 8 in 2024-03-24T11:49:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:21 smithi119 ceph-mon[33714]: osdmap e427: 8 total, 8 up, 8 in 2024-03-24T11:49:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:21.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:21 smithi119 ceph-mon[33714]: osdmap e428: 8 total, 8 up, 8 in 2024-03-24T11:49:22.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:22 smithi119 ceph-mon[33714]: pgmap v641: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:22.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:22.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:22 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:22.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:49:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:49:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:22 smithi114 ceph-mon[30312]: pgmap v641: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:22 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:22 smithi114 ceph-mon[42466]: pgmap v641: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:22 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:23 smithi114 ceph-mon[30312]: osdmap e429: 8 total, 8 up, 8 in 2024-03-24T11:49:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:23 smithi114 ceph-mon[42466]: osdmap e429: 8 total, 8 up, 8 in 2024-03-24T11:49:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:23 smithi119 ceph-mon[33714]: osdmap e429: 8 total, 8 up, 8 in 2024-03-24T11:49:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[30312]: pgmap v644: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[30312]: osdmap e430: 8 total, 8 up, 8 in 2024-03-24T11:49:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-75"}]: dispatch 2024-03-24T11:49:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-75"}]: dispatch 2024-03-24T11:49:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[42466]: pgmap v644: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[42466]: osdmap e430: 8 total, 8 up, 8 in 2024-03-24T11:49:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-75"}]: dispatch 2024-03-24T11:49:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-75"}]: dispatch 2024-03-24T11:49:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:24 smithi119 ceph-mon[33714]: pgmap v644: 356 pgs: 356 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:24 smithi119 ceph-mon[33714]: osdmap e430: 8 total, 8 up, 8 in 2024-03-24T11:49:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-75"}]: dispatch 2024-03-24T11:49:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-75"}]: dispatch 2024-03-24T11:49:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:49:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:49:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:49:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:25 smithi114 ceph-mon[30312]: osdmap e431: 8 total, 8 up, 8 in 2024-03-24T11:49:25.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:25 smithi114 ceph-mon[42466]: osdmap e431: 8 total, 8 up, 8 in 2024-03-24T11:49:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:25 smithi119 ceph-mon[33714]: osdmap e431: 8 total, 8 up, 8 in 2024-03-24T11:49:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[42466]: pgmap v647: 324 pgs: 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:49:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[42466]: osdmap e432: 8 total, 8 up, 8 in 2024-03-24T11:49:26.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:26.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:26.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[30312]: pgmap v647: 324 pgs: 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:49:26.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[30312]: osdmap e432: 8 total, 8 up, 8 in 2024-03-24T11:49:26.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:26.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:26 smithi119 ceph-mon[33714]: pgmap v647: 324 pgs: 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:49:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:26 smithi119 ceph-mon[33714]: osdmap e432: 8 total, 8 up, 8 in 2024-03-24T11:49:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:27.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:27.534 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:27 smithi114 ceph-mon[42466]: osdmap e433: 8 total, 8 up, 8 in 2024-03-24T11:49:27.534 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:27.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:27.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:27 smithi114 ceph-mon[30312]: osdmap e433: 8 total, 8 up, 8 in 2024-03-24T11:49:27.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:27 smithi119 ceph-mon[33714]: osdmap e433: 8 total, 8 up, 8 in 2024-03-24T11:49:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[42466]: pgmap v650: 356 pgs: 356 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:49:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[42466]: osdmap e434: 8 total, 8 up, 8 in 2024-03-24T11:49:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-77"}]: dispatch 2024-03-24T11:49:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-77"}]: dispatch 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[30312]: pgmap v650: 356 pgs: 356 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[30312]: osdmap e434: 8 total, 8 up, 8 in 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-77"}]: dispatch 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-77"}]: dispatch 2024-03-24T11:49:28.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:28 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:28 smithi119 ceph-mon[33714]: pgmap v650: 356 pgs: 356 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:49:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:28 smithi119 ceph-mon[33714]: osdmap e434: 8 total, 8 up, 8 in 2024-03-24T11:49:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-77"}]: dispatch 2024-03-24T11:49:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-77"}]: dispatch 2024-03-24T11:49:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:28 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:29.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:29 smithi114 ceph-mon[42466]: osdmap e435: 8 total, 8 up, 8 in 2024-03-24T11:49:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:29 smithi114 ceph-mon[30312]: osdmap e435: 8 total, 8 up, 8 in 2024-03-24T11:49:29.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:29 smithi119 ceph-mon[33714]: osdmap e435: 8 total, 8 up, 8 in 2024-03-24T11:49:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[42466]: pgmap v653: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:49:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[42466]: osdmap e436: 8 total, 8 up, 8 in 2024-03-24T11:49:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[30312]: pgmap v653: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:49:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[30312]: osdmap e436: 8 total, 8 up, 8 in 2024-03-24T11:49:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:30.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:30 smithi119 ceph-mon[33714]: pgmap v653: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:49:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:30 smithi119 ceph-mon[33714]: osdmap e436: 8 total, 8 up, 8 in 2024-03-24T11:49:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:31.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:31 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:31.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:31 smithi114 ceph-mon[42466]: osdmap e437: 8 total, 8 up, 8 in 2024-03-24T11:49:31.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:31 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:31 smithi114 ceph-mon[30312]: osdmap e437: 8 total, 8 up, 8 in 2024-03-24T11:49:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:31 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:31 smithi119 ceph-mon[33714]: osdmap e437: 8 total, 8 up, 8 in 2024-03-24T11:49:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:32.212 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:49:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:49:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:32 smithi114 ceph-mon[42466]: pgmap v656: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:32.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:32 smithi114 ceph-mon[42466]: osdmap e438: 8 total, 8 up, 8 in 2024-03-24T11:49:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:32 smithi114 ceph-mon[30312]: pgmap v656: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:32 smithi114 ceph-mon[30312]: osdmap e438: 8 total, 8 up, 8 in 2024-03-24T11:49:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:32 smithi119 ceph-mon[33714]: pgmap v656: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:49:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:32 smithi119 ceph-mon[33714]: osdmap e438: 8 total, 8 up, 8 in 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[30312]: osdmap e439: 8 total, 8 up, 8 in 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "275.8"}]: dispatch 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "275.8"}]: dispatch 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[30312]: 275.8 deep-scrub starts 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[30312]: 275.8 deep-scrub ok 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[42466]: osdmap e439: 8 total, 8 up, 8 in 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "275.8"}]: dispatch 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "275.8"}]: dispatch 2024-03-24T11:49:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[42466]: 275.8 deep-scrub starts 2024-03-24T11:49:33.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:33 smithi114 ceph-mon[42466]: 275.8 deep-scrub ok 2024-03-24T11:49:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:33 smithi119 ceph-mon[33714]: osdmap e439: 8 total, 8 up, 8 in 2024-03-24T11:49:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "275.8"}]: dispatch 2024-03-24T11:49:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:33 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.119:0/4055882710' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "275.8"}]: dispatch 2024-03-24T11:49:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:33 smithi119 ceph-mon[33714]: 275.8 deep-scrub starts 2024-03-24T11:49:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:33 smithi119 ceph-mon[33714]: 275.8 deep-scrub ok 2024-03-24T11:49:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:34 smithi114 ceph-mon[30312]: pgmap v659: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-24T11:49:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:34 smithi114 ceph-mon[42466]: pgmap v659: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-24T11:49:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:34 smithi119 ceph-mon[33714]: pgmap v659: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-24T11:49:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:49:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:49:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:35.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:49:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:49:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:36 smithi114 ceph-mon[30312]: pgmap v660: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 902 B/s wr, 128 op/s 2024-03-24T11:49:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:36 smithi114 ceph-mon[42466]: pgmap v660: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 902 B/s wr, 128 op/s 2024-03-24T11:49:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:36 smithi119 ceph-mon[33714]: pgmap v660: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 902 B/s wr, 128 op/s 2024-03-24T11:49:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:38 smithi114 ceph-mon[30312]: pgmap v661: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 921 B/s wr, 110 op/s 2024-03-24T11:49:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:38 smithi114 ceph-mon[42466]: pgmap v661: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 921 B/s wr, 110 op/s 2024-03-24T11:49:38.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:38 smithi119 ceph-mon[33714]: pgmap v661: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 921 B/s wr, 110 op/s 2024-03-24T11:49:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:40 smithi114 ceph-mon[30312]: pgmap v662: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 767 B/s wr, 92 op/s 2024-03-24T11:49:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:40 smithi114 ceph-mon[42466]: pgmap v662: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 767 B/s wr, 92 op/s 2024-03-24T11:49:40.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:40 smithi119 ceph-mon[33714]: pgmap v662: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 767 B/s wr, 92 op/s 2024-03-24T11:49:42.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:49:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:49:42.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:42 smithi119 ceph-mon[33714]: pgmap v663: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 636 B/s wr, 76 op/s 2024-03-24T11:49:42.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:42 smithi114 ceph-mon[30312]: pgmap v663: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 636 B/s wr, 76 op/s 2024-03-24T11:49:42.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:42 smithi114 ceph-mon[42466]: pgmap v663: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 636 B/s wr, 76 op/s 2024-03-24T11:49:43.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:43.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:43.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:44 smithi119 ceph-mon[33714]: pgmap v664: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 96 B/s wr, 1 op/s 2024-03-24T11:49:44.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:44 smithi114 ceph-mon[30312]: pgmap v664: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 96 B/s wr, 1 op/s 2024-03-24T11:49:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:44 smithi114 ceph-mon[42466]: pgmap v664: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 96 B/s wr, 1 op/s 2024-03-24T11:49:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:49:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:49:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:49:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:46 smithi119 ceph-mon[33714]: pgmap v665: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:49:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:46 smithi114 ceph-mon[30312]: pgmap v665: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:49:46.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:46 smithi114 ceph-mon[42466]: pgmap v665: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:49:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:48 smithi119 ceph-mon[33714]: pgmap v666: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:49:48.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:48 smithi114 ceph-mon[30312]: pgmap v666: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:49:48.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:48 smithi114 ceph-mon[42466]: pgmap v666: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:49:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:50 smithi119 ceph-mon[33714]: pgmap v667: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:50 smithi114 ceph-mon[30312]: pgmap v667: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:50 smithi114 ceph-mon[42466]: pgmap v667: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:52.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:49:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:49:52.554 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 11:49:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T11:49:52.256+0000 7fe34dccc700 -1 snap_mapper.add_oid found existing snaps mapped on 275:143a67ab:test-rados-api-smithi114-82677-80::foo:2, removing 2024-03-24T11:49:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:52 smithi119 ceph-mon[33714]: pgmap v668: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:52.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:52.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:52.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-79"}]: dispatch 2024-03-24T11:49:52.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:52 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-79"}]: dispatch 2024-03-24T11:49:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[30312]: pgmap v668: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-79"}]: dispatch 2024-03-24T11:49:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-79"}]: dispatch 2024-03-24T11:49:52.739 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3[65524]: 2024-03-24T11:49:52.256+0000 7fdb4845e700 -1 snap_mapper.add_oid found existing snaps mapped on 275:143a67ab:test-rados-api-smithi114-82677-80::foo:2, removing 2024-03-24T11:49:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[42466]: pgmap v668: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-79"}]: dispatch 2024-03-24T11:49:52.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-79"}]: dispatch 2024-03-24T11:49:52.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 11:49:52 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0[46853]: 2024-03-24T11:49:52.256+0000 7f2fad6bb700 -1 snap_mapper.add_oid found existing snaps mapped on 275:143a67ab:test-rados-api-smithi114-82677-80::foo:2, removing 2024-03-24T11:49:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:54 smithi119 ceph-mon[33714]: pgmap v669: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:54 smithi119 ceph-mon[33714]: osdmap e440: 8 total, 8 up, 8 in 2024-03-24T11:49:54.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:54 smithi114 ceph-mon[30312]: pgmap v669: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:54.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:54 smithi114 ceph-mon[30312]: osdmap e440: 8 total, 8 up, 8 in 2024-03-24T11:49:54.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:54 smithi114 ceph-mon[42466]: pgmap v669: 356 pgs: 356 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:54.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:54 smithi114 ceph-mon[42466]: osdmap e440: 8 total, 8 up, 8 in 2024-03-24T11:49:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:49:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:49:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:49:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:55 smithi119 ceph-mon[33714]: osdmap e441: 8 total, 8 up, 8 in 2024-03-24T11:49:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:55 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:55.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:55 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:55 smithi114 ceph-mon[30312]: osdmap e441: 8 total, 8 up, 8 in 2024-03-24T11:49:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:55 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:55.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:55 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:55 smithi114 ceph-mon[42466]: osdmap e441: 8 total, 8 up, 8 in 2024-03-24T11:49:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:55 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:55 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:56 smithi119 ceph-mon[33714]: pgmap v672: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:49:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:56 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:56 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:56 smithi119 ceph-mon[33714]: osdmap e442: 8 total, 8 up, 8 in 2024-03-24T11:49:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:49:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:56 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:49:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[42466]: pgmap v672: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:49:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[42466]: osdmap e442: 8 total, 8 up, 8 in 2024-03-24T11:49:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:49:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:49:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[30312]: pgmap v672: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:49:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:49:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[30312]: osdmap e442: 8 total, 8 up, 8 in 2024-03-24T11:49:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:49:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:49:56.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:56 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:49:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:49:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:57 smithi119 ceph-mon[33714]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T11:49:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:57 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:57.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:49:57.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[42466]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T11:49:57.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:57.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:49:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[30312]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T11:49:57.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:57.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:57 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:49:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[30312]: pgmap v675: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-24T11:49:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:49:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:49:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[30312]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T11:49:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:49:58.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[42466]: pgmap v675: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-24T11:49:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:49:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:49:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[42466]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T11:49:58.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:58 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:49:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:58 smithi119 ceph-mon[33714]: pgmap v675: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-24T11:49:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:49:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:49:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:58 smithi119 ceph-mon[33714]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T11:49:58.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:58 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:49:59.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[42466]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[30312]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:49:59.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:49:59 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:49:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:49:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:59 smithi119 ceph-mon[33714]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T11:49:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:49:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:49:59.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:49:59 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: pgmap v678: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: Health detail: HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-24T11:50:00.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: application not enabled on pool 'SetOpFlagssmithi114-83748-1' 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: application not enabled on pool 'WatchNotifysmithi114-83760-1' 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: application not enabled on pool 'SnapListsmithi114-83801-1' 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: application not enabled on pool 'AssertExistssmithi114-83830-1' 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[42466]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: pgmap v678: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: Health detail: HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-24T11:50:00.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: application not enabled on pool 'SetOpFlagssmithi114-83748-1' 2024-03-24T11:50:00.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: application not enabled on pool 'WatchNotifysmithi114-83760-1' 2024-03-24T11:50:00.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: application not enabled on pool 'SnapListsmithi114-83801-1' 2024-03-24T11:50:00.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: application not enabled on pool 'AssertExistssmithi114-83830-1' 2024-03-24T11:50:00.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:00 smithi114 ceph-mon[30312]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: pgmap v678: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: Health detail: HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-24T11:50:00.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: application not enabled on pool 'SetOpFlagssmithi114-83748-1' 2024-03-24T11:50:00.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: application not enabled on pool 'WatchNotifysmithi114-83760-1' 2024-03-24T11:50:00.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: application not enabled on pool 'SnapListsmithi114-83801-1' 2024-03-24T11:50:00.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: application not enabled on pool 'AssertExistssmithi114-83830-1' 2024-03-24T11:50:00.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:00 smithi119 ceph-mon[33714]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-24T11:50:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-24T11:50:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[30312]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T11:50:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-24T11:50:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-24T11:50:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-24T11:50:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[42466]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T11:50:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-24T11:50:01.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:01 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:01 smithi119 ceph-mon[33714]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-24T11:50:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:01 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-24T11:50:02.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:50:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:50:02.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[30312]: pgmap v681: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:02.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-24T11:50:02.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[30312]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T11:50:02.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[42466]: pgmap v681: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-24T11:50:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[42466]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T11:50:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:02.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:02 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:02 smithi119 ceph-mon[33714]: pgmap v681: 356 pgs: 356 active+clean; 8.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-24T11:50:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:02 smithi119 ceph-mon[33714]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T11:50:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:02 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:03.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-81"}]: dispatch 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-81"}]: dispatch 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T11:50:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-81"}]: dispatch 2024-03-24T11:50:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-81"}]: dispatch 2024-03-24T11:50:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:03 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T11:50:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-81"}]: dispatch 2024-03-24T11:50:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-81"}]: dispatch 2024-03-24T11:50:03.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:03 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:04 smithi114 ceph-mon[30312]: pgmap v684: 356 pgs: 356 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-24T11:50:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:04 smithi114 ceph-mon[30312]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T11:50:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:04 smithi114 ceph-mon[42466]: pgmap v684: 356 pgs: 356 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-24T11:50:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:04 smithi114 ceph-mon[42466]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T11:50:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:04 smithi119 ceph-mon[33714]: pgmap v684: 356 pgs: 356 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-24T11:50:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:04 smithi119 ceph-mon[33714]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T11:50:04.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:50:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:50:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[30312]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[30312]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[42466]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:05.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:05.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:05 smithi114 ceph-mon[42466]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T11:50:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:05 smithi119 ceph-mon[33714]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T11:50:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:05 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:05 smithi119 ceph-mon[33714]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: pgmap v687: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: pgmap v687: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-24T11:50:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T11:50:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:06.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: pgmap v687: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[30312]: pgmap v690: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T11:50:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:50:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[30312]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T11:50:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[42466]: pgmap v690: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T11:50:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:50:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[42466]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T11:50:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:08.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:08 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:08 smithi119 ceph-mon[33714]: pgmap v690: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T11:50:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:50:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:08 smithi119 ceph-mon[33714]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T11:50:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:08 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[42466]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[42466]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[30312]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:09.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:09.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:09 smithi114 ceph-mon[30312]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T11:50:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:09 smithi119 ceph-mon[33714]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T11:50:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:09.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:09 smithi119 ceph-mon[33714]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T11:50:10.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:10 smithi114 ceph-mon[30312]: pgmap v693: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-24T11:50:10.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:10 smithi114 ceph-mon[42466]: pgmap v693: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-24T11:50:10.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:10 smithi119 ceph-mon[33714]: pgmap v693: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-24T11:50:11.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:11 smithi114 ceph-mon[30312]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T11:50:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:11 smithi114 ceph-mon[42466]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T11:50:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:11 smithi119 ceph-mon[33714]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T11:50:12.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:50:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:50:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[30312]: pgmap v696: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[30312]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T11:50:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-83"}]: dispatch 2024-03-24T11:50:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-83"}]: dispatch 2024-03-24T11:50:12.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[42466]: pgmap v696: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:12.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[42466]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T11:50:12.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:12.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:12.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-83"}]: dispatch 2024-03-24T11:50:12.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:12 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-83"}]: dispatch 2024-03-24T11:50:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:12 smithi119 ceph-mon[33714]: pgmap v696: 356 pgs: 356 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:12 smithi119 ceph-mon[33714]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T11:50:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-83"}]: dispatch 2024-03-24T11:50:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:12 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-83"}]: dispatch 2024-03-24T11:50:13.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:13.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:13 smithi114 ceph-mon[30312]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T11:50:13.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:13.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:13 smithi114 ceph-mon[42466]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T11:50:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:13 smithi119 ceph-mon[33714]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T11:50:14.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:50:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:14.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[42466]: pgmap v699: 324 pgs: 324 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-24T11:50:14.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[42466]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T11:50:14.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:14.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:14.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[30312]: pgmap v699: 324 pgs: 324 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-24T11:50:14.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[30312]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T11:50:14.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:14.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:14 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:14 smithi119 ceph-mon[33714]: pgmap v699: 324 pgs: 324 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-24T11:50:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:14 smithi119 ceph-mon[33714]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T11:50:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:14 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[30312]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[42466]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:15 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:15 smithi119 ceph-mon[33714]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T11:50:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:15 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[30312]: pgmap v702: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[30312]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[42466]: pgmap v702: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[42466]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T11:50:16.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:16 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:16 smithi119 ceph-mon[33714]: pgmap v702: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-24T11:50:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:16 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:16 smithi119 ceph-mon[33714]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T11:50:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:16 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:17.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[30312]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[30312]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[42466]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:17.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:17.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[42466]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T11:50:17.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_tier","val": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:50:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:17 smithi119 ceph-mon[33714]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T11:50:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:17 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:17 smithi119 ceph-mon[33714]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T11:50:17.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[30312]: pgmap v705: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[30312]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T11:50:18.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[42466]: pgmap v705: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:18.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:18.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:18 smithi114 ceph-mon[42466]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T11:50:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:18 smithi119 ceph-mon[33714]: pgmap v705: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:18 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:18 smithi119 ceph-mon[33714]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T11:50:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:19.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:19.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:20 smithi114 ceph-mon[30312]: pgmap v708: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:20 smithi114 ceph-mon[30312]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T11:50:20.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:20 smithi114 ceph-mon[42466]: pgmap v708: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:20.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:20 smithi114 ceph-mon[42466]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T11:50:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:20 smithi119 ceph-mon[33714]: pgmap v708: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:20 smithi119 ceph-mon[33714]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[42466]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-85"}]: dispatch 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-85"}]: dispatch 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[30312]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:21.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-85"}]: dispatch 2024-03-24T11:50:21.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-85"}]: dispatch 2024-03-24T11:50:21.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:21 smithi119 ceph-mon[33714]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T11:50:21.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:21.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:21.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-85"}]: dispatch 2024-03-24T11:50:21.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-85"}]: dispatch 2024-03-24T11:50:22.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:50:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:50:22.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:22 smithi114 ceph-mon[30312]: pgmap v711: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:22 smithi114 ceph-mon[30312]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T11:50:22.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:22 smithi114 ceph-mon[42466]: pgmap v711: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:22.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:22 smithi114 ceph-mon[42466]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T11:50:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:22 smithi119 ceph-mon[33714]: pgmap v711: 356 pgs: 356 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:22 smithi119 ceph-mon[33714]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T11:50:23.503 INFO:tasks.workunit.client.0.smithi114.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18111 ms) 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22143 ms) 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3016 ms) 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3028 ms) 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3023 ms) 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3034 ms) 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.504 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4029 ms) 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38140 ms) 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16394 ms) 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4031 ms) 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3041 ms) 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25083 ms) 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-24T11:50:23.505 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15048 ms) 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5044 ms) 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4041 ms) 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: waiting for scrubs... 2024-03-24T11:50:23.506 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: done waiting 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25081 ms) 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10074 ms) 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9061 ms) 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9068 ms) 2024-03-24T11:50:23.507 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-24T11:50:23.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[30312]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T11:50:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:23.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[42466]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T11:50:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:23 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:23.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:23 smithi119 ceph-mon[33714]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T11:50:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:23.812 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:23 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:24.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:50:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:24.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[30312]: pgmap v714: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-24T11:50:24.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:24.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:24.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[30312]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T11:50:24.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[42466]: pgmap v714: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-24T11:50:24.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:24.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:24.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[42466]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T11:50:24.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:24 smithi119 ceph-mon[33714]: pgmap v714: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-24T11:50:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:24 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:24 smithi119 ceph-mon[33714]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T11:50:24.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:25 smithi119 ceph-mon[33714]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T11:50:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:25.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:25 smithi114 ceph-mon[42466]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T11:50:25.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:25.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:25 smithi114 ceph-mon[30312]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T11:50:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:25.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:26 smithi119 ceph-mon[33714]: pgmap v717: 388 pgs: 64 unknown, 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-24T11:50:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]: dispatch 2024-03-24T11:50:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:26 smithi119 ceph-mon[33714]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T11:50:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]: dispatch 2024-03-24T11:50:26.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[30312]: pgmap v717: 388 pgs: 64 unknown, 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-24T11:50:26.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:26.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]: dispatch 2024-03-24T11:50:26.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[30312]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T11:50:26.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]: dispatch 2024-03-24T11:50:26.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[42466]: pgmap v717: 388 pgs: 64 unknown, 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-24T11:50:26.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:26.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]: dispatch 2024-03-24T11:50:26.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[42466]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T11:50:26.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]: dispatch 2024-03-24T11:50:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]': finished 2024-03-24T11:50:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:27 smithi119 ceph-mon[33714]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T11:50:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]': finished 2024-03-24T11:50:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[30312]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T11:50:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:27.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:27.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_tier","val": "test-rados-api-smithi114-82677-89-test-flush"}]': finished 2024-03-24T11:50:27.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[42466]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T11:50:27.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:27.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:50:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:28 smithi119 ceph-mon[33714]: pgmap v720: 388 pgs: 388 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:28 smithi119 ceph-mon[33714]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T11:50:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:28 smithi119 ceph-mon[33714]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[30312]: pgmap v720: 388 pgs: 388 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[30312]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[30312]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[42466]: pgmap v720: 388 pgs: 388 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:28.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:50:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[42466]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T11:50:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:50:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:50:28.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:28 smithi114 ceph-mon[42466]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T11:50:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:30 smithi119 ceph-mon[33714]: pgmap v723: 388 pgs: 388 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:30 smithi119 ceph-mon[33714]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T11:50:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-87"}]: dispatch 2024-03-24T11:50:30.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-87"}]: dispatch 2024-03-24T11:50:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[42466]: pgmap v723: 388 pgs: 388 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[42466]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-87"}]: dispatch 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-87"}]: dispatch 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[30312]: pgmap v723: 388 pgs: 388 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[30312]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T11:50:30.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-87"}]: dispatch 2024-03-24T11:50:30.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-87"}]: dispatch 2024-03-24T11:50:31.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:31 smithi119 ceph-mon[33714]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T11:50:31.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:31 smithi114 ceph-mon[30312]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T11:50:31.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:31 smithi114 ceph-mon[42466]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T11:50:32.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:50:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:50:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:32 smithi119 ceph-mon[33714]: pgmap v726: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:32 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:32 smithi119 ceph-mon[33714]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T11:50:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:32.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:32 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:32.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[42466]: pgmap v726: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:32.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:32.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[42466]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T11:50:32.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:32.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:32.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[30312]: pgmap v726: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:32.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:32.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[30312]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T11:50:32.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:32.891 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:32 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:33.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:33 smithi119 ceph-mon[33714]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T11:50:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:33.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[42466]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[30312]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:33.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:50:34.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:50:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[30312]: pgmap v729: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:50:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[30312]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T11:50:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[42466]: pgmap v729: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:50:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[42466]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T11:50:34.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:34 smithi119 ceph-mon[33714]: pgmap v729: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:50:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:34 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:50:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:34 smithi119 ceph-mon[33714]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T11:50:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:35.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:35 smithi114 ceph-mon[42466]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T11:50:35.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:35 smithi114 ceph-mon[42466]: pgmap v732: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:50:35.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:35.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:35 smithi114 ceph-mon[30312]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T11:50:35.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:35 smithi114 ceph-mon[30312]: pgmap v732: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:50:35.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:36.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:35 smithi119 ceph-mon[33714]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T11:50:36.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:35 smithi119 ceph-mon[33714]: pgmap v732: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:50:36.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:50:36.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T11:50:36.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:36.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:36.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-90"}]: dispatch 2024-03-24T11:50:36.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-90"}]: dispatch 2024-03-24T11:50:36.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi119 to 3248M 2024-03-24T11:50:36.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: Adjusting osd_memory_target on smithi114 to 3248M 2024-03-24T11:50:36.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:36.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:36.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:36.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:36.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:36.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T11:50:36.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:36.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:36.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-90"}]: dispatch 2024-03-24T11:50:36.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-90"}]: dispatch 2024-03-24T11:50:36.993 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:36.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi119 to 3248M 2024-03-24T11:50:36.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: Adjusting osd_memory_target on smithi114 to 3248M 2024-03-24T11:50:36.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:36.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:36.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:36.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:36.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:37.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T11:50:37.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:37.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:37.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-90"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-90"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi119 to 3248M 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: Adjusting osd_memory_target on smithi114 to 3248M 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:37.055 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:37.056 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:50:37.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:37 smithi114 ceph-mon[30312]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T11:50:37.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:37 smithi114 ceph-mon[30312]: pgmap v735: 324 pgs: 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-24T11:50:37.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:37 smithi114 ceph-mon[42466]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T11:50:37.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:37 smithi114 ceph-mon[42466]: pgmap v735: 324 pgs: 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-24T11:50:38.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:37 smithi119 ceph-mon[33714]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T11:50:38.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:37 smithi119 ceph-mon[33714]: pgmap v735: 324 pgs: 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-24T11:50:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:38 smithi114 ceph-mon[42466]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T11:50:38.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:38.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:38.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:38 smithi114 ceph-mon[30312]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T11:50:38.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:38.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:39.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:38 smithi119 ceph-mon[33714]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T11:50:39.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:39.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[42466]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[42466]: pgmap v738: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[30312]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:39.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:39 smithi114 ceph-mon[30312]: pgmap v738: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-24T11:50:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:39 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:39 smithi119 ceph-mon[33714]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T11:50:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:40.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:39 smithi119 ceph-mon[33714]: pgmap v738: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-24T11:50:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:40 smithi114 ceph-mon[30312]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T11:50:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:40 smithi114 ceph-mon[42466]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T11:50:41.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:40 smithi119 ceph-mon[33714]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T11:50:41.963 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:41 smithi119 ceph-mon[33714]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T11:50:41.963 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:41.964 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:41.964 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-92"}]: dispatch 2024-03-24T11:50:41.964 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-92"}]: dispatch 2024-03-24T11:50:41.964 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:41 smithi119 ceph-mon[33714]: pgmap v741: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:41.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[42466]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T11:50:41.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-92"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-92"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[42466]: pgmap v741: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[30312]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-92"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-92"}]: dispatch 2024-03-24T11:50:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:41 smithi114 ceph-mon[30312]: pgmap v741: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:50:42.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:50:41 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:50:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:42 smithi114 ceph-mon[30312]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T11:50:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:42 smithi114 ceph-mon[42466]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T11:50:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:42 smithi119 ceph-mon[33714]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T11:50:43.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[42466]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[42466]: pgmap v744: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[30312]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:43.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:43 smithi114 ceph-mon[30312]: pgmap v744: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:50:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:43 smithi119 ceph-mon[33714]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T11:50:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:50:44.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:43 smithi119 ceph-mon[33714]: pgmap v744: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[42466]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[30312]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:44.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:50:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:50:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:44 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:44 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:50:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:44 smithi119 ceph-mon[33714]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T11:50:45.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:50:45.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:45 smithi114 ceph-mon[30312]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T11:50:45.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:45 smithi114 ceph-mon[30312]: pgmap v747: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:50:45.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:45 smithi114 ceph-mon[42466]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T11:50:45.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:45 smithi114 ceph-mon[42466]: pgmap v747: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:50:46.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:45 smithi119 ceph-mon[33714]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T11:50:46.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:45 smithi119 ceph-mon[33714]: pgmap v747: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T11:50:46.640 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:50:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=cleanup t=2024-03-24T11:50:46.328358282Z level=info msg="Completed cleanup jobs" duration=2.473378ms 2024-03-24T11:50:46.640 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:50:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=sqlstore.transactions t=2024-03-24T11:50:46.336516046Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:50:46.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:46 smithi114 ceph-mon[30312]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T11:50:46.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:46 smithi114 ceph-mon[42466]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T11:50:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:46 smithi119 ceph-mon[33714]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T11:50:47.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:47 smithi114 ceph-mon[30312]: pgmap v749: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T11:50:47.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:47 smithi114 ceph-mon[42466]: pgmap v749: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T11:50:48.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:47 smithi119 ceph-mon[33714]: pgmap v749: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T11:50:48.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:48.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:49.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:49 smithi114 ceph-mon[30312]: pgmap v750: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-24T11:50:49.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:49 smithi114 ceph-mon[42466]: pgmap v750: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-24T11:50:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:49 smithi119 ceph-mon[33714]: pgmap v750: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-24T11:50:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:51 smithi114 ceph-mon[30312]: pgmap v751: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T11:50:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:51 smithi114 ceph-mon[42466]: pgmap v751: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T11:50:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:51 smithi119 ceph-mon[33714]: pgmap v751: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T11:50:52.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:50:51 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:50:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:52 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:52 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:52 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:52 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:52 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:52 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:50:54.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:53 smithi114 ceph-mon[30312]: pgmap v752: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 992 B/s wr, 4 op/s 2024-03-24T11:50:54.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:53 smithi114 ceph-mon[42466]: pgmap v752: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 992 B/s wr, 4 op/s 2024-03-24T11:50:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:53 smithi119 ceph-mon[33714]: pgmap v752: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 992 B/s wr, 4 op/s 2024-03-24T11:50:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:50:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:50:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:56.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:55 smithi114 ceph-mon[30312]: pgmap v753: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-24T11:50:56.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:55 smithi114 ceph-mon[42466]: pgmap v753: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-24T11:50:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:55 smithi119 ceph-mon[33714]: pgmap v753: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-24T11:50:57.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:56 smithi114 ceph-mon[30312]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T11:50:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:56 smithi114 ceph-mon[42466]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T11:50:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:56 smithi119 ceph-mon[33714]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T11:50:58.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:57 smithi114 ceph-mon[30312]: pgmap v755: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:50:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:57 smithi114 ceph-mon[30312]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T11:50:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:57 smithi114 ceph-mon[42466]: pgmap v755: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:50:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:57 smithi114 ceph-mon[42466]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T11:50:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:57 smithi119 ceph-mon[33714]: pgmap v755: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:50:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:57 smithi119 ceph-mon[33714]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T11:51:00.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:50:59 smithi114 ceph-mon[30312]: pgmap v757: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:50:59 smithi114 ceph-mon[42466]: pgmap v757: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:00.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:50:59 smithi119 ceph-mon[33714]: pgmap v757: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:02.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:01 smithi114 ceph-mon[30312]: pgmap v758: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:02.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:01 smithi114 ceph-mon[42466]: pgmap v758: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:02.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:01 smithi119 ceph-mon[33714]: pgmap v758: 356 pgs: 356 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:02.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:51:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:51:03.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:02 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:03.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:02 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:03.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:02 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:04.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:03 smithi119 ceph-mon[33714]: pgmap v759: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:04.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:03 smithi114 ceph-mon[30312]: pgmap v759: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:03 smithi114 ceph-mon[42466]: pgmap v759: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:04.990 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:51:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:51:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:51:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:05 smithi119 ceph-mon[33714]: pgmap v760: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:05 smithi114 ceph-mon[30312]: pgmap v760: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:05 smithi114 ceph-mon[42466]: pgmap v760: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-94"}]: dispatch 2024-03-24T11:51:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:06 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-94"}]: dispatch 2024-03-24T11:51:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-94"}]: dispatch 2024-03-24T11:51:07.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-94"}]: dispatch 2024-03-24T11:51:07.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:07.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:07.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-94"}]: dispatch 2024-03-24T11:51:07.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:06 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-94"}]: dispatch 2024-03-24T11:51:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:08 smithi119 ceph-mon[33714]: pgmap v761: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-24T11:51:08.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:08 smithi119 ceph-mon[33714]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T11:51:08.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:08 smithi114 ceph-mon[30312]: pgmap v761: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-24T11:51:08.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:08 smithi114 ceph-mon[30312]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T11:51:08.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:08 smithi114 ceph-mon[42466]: pgmap v761: 356 pgs: 356 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-24T11:51:08.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:08 smithi114 ceph-mon[42466]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T11:51:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:09 smithi119 ceph-mon[33714]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T11:51:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:09 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:09 smithi119 ceph-mon[33714]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[30312]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[30312]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[42466]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:09.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:09.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:09 smithi114 ceph-mon[42466]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T11:51:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:10 smithi119 ceph-mon[33714]: pgmap v764: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:51:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:10 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:10 smithi119 ceph-mon[33714]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[30312]: pgmap v764: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[30312]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[42466]: pgmap v764: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:10 smithi114 ceph-mon[42466]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T11:51:12.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:12 smithi119 ceph-mon[33714]: pgmap v767: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:12.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:51:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:51:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:12 smithi114 ceph-mon[30312]: pgmap v767: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:12 smithi114 ceph-mon[42466]: pgmap v767: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:13.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:14.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:14 smithi119 ceph-mon[33714]: pgmap v768: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 2 op/s 2024-03-24T11:51:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:14 smithi114 ceph-mon[30312]: pgmap v768: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 2 op/s 2024-03-24T11:51:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:14 smithi114 ceph-mon[42466]: pgmap v768: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 2 op/s 2024-03-24T11:51:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:51:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:51:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:51:16.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:16 smithi119 ceph-mon[33714]: pgmap v769: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 741 B/s wr, 2 op/s 2024-03-24T11:51:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:16 smithi114 ceph-mon[42466]: pgmap v769: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 741 B/s wr, 2 op/s 2024-03-24T11:51:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:16 smithi114 ceph-mon[30312]: pgmap v769: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 741 B/s wr, 2 op/s 2024-03-24T11:51:18.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:18 smithi119 ceph-mon[33714]: pgmap v770: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-03-24T11:51:18.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:18 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:18 smithi114 ceph-mon[30312]: pgmap v770: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-03-24T11:51:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:18 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:18 smithi114 ceph-mon[42466]: pgmap v770: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-03-24T11:51:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:18 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:19.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:19.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:20 smithi119 ceph-mon[33714]: pgmap v771: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 517 B/s wr, 2 op/s 2024-03-24T11:51:20.304 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 11:51:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T11:51:20.037+0000 7f40a7e08700 -1 snap_mapper.add_oid found existing snaps mapped on 97:ecfc96d5:test-rados-api-smithi114-82677-97::foo:21, removing 2024-03-24T11:51:20.305 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 11:51:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T11:51:20.037+0000 7fe34a4c5700 -1 snap_mapper.add_oid found existing snaps mapped on 97:ecfc96d5:test-rados-api-smithi114-82677-97::foo:21, removing 2024-03-24T11:51:20.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:20 smithi114 ceph-mon[30312]: pgmap v771: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 517 B/s wr, 2 op/s 2024-03-24T11:51:20.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:20 smithi114 ceph-mon[42466]: pgmap v771: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 517 B/s wr, 2 op/s 2024-03-24T11:51:20.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 11:51:20 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1[52784]: 2024-03-24T11:51:20.037+0000 7f0aba5ac700 -1 snap_mapper.add_oid found existing snaps mapped on 97:ecfc96d5:test-rados-api-smithi114-82677-97::foo:21, removing 2024-03-24T11:51:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-96"}]: dispatch 2024-03-24T11:51:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:21 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-96"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-96"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-96"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-96"}]: dispatch 2024-03-24T11:51:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:21 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-96"}]: dispatch 2024-03-24T11:51:22.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:51:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:51:22.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:22 smithi119 ceph-mon[33714]: pgmap v772: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 939 B/s rd, 469 B/s wr, 1 op/s 2024-03-24T11:51:22.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:22 smithi119 ceph-mon[33714]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T11:51:22.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:22 smithi114 ceph-mon[42466]: pgmap v772: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 939 B/s rd, 469 B/s wr, 1 op/s 2024-03-24T11:51:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:22 smithi114 ceph-mon[42466]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T11:51:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:22 smithi114 ceph-mon[30312]: pgmap v772: 356 pgs: 356 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 939 B/s rd, 469 B/s wr, 1 op/s 2024-03-24T11:51:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:22 smithi114 ceph-mon[30312]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T11:51:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[42466]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T11:51:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[30312]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T11:51:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:23.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:23 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:23 smithi119 ceph-mon[33714]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T11:51:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:23 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[30312]: pgmap v775: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[30312]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[42466]: pgmap v775: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:24.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[42466]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T11:51:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:24.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:24 smithi119 ceph-mon[33714]: pgmap v775: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-03-24T11:51:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:24 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:24 smithi119 ceph-mon[33714]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T11:51:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:24 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:51:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:51:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:51:25.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:51:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]: dispatch 2024-03-24T11:51:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[30312]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T11:51:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]: dispatch 2024-03-24T11:51:25.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:51:25.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]: dispatch 2024-03-24T11:51:25.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[42466]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T11:51:25.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:25 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]: dispatch 2024-03-24T11:51:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:51:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]: dispatch 2024-03-24T11:51:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:25 smithi119 ceph-mon[33714]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T11:51:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:25 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]: dispatch 2024-03-24T11:51:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[42466]: pgmap v778: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-24T11:51:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]': finished 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[42466]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[30312]: pgmap v778: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]': finished 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[30312]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T11:51:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:26 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:26 smithi119 ceph-mon[33714]: pgmap v778: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-24T11:51:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:26 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:51:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-98", "mode": "writeback"}]': finished 2024-03-24T11:51:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:26 smithi119 ceph-mon[33714]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T11:51:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:26 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]': finished 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[30312]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[30312]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T11:51:27.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]': finished 2024-03-24T11:51:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[42466]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T11:51:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:51:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:51:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:51:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:51:27.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:27 smithi114 ceph-mon[42466]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T11:51:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-98"}]': finished 2024-03-24T11:51:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:27 smithi119 ceph-mon[33714]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T11:51:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:51:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:51:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:27 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:51:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:27 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:51:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:27 smithi119 ceph-mon[33714]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T11:51:28.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[30312]: pgmap v781: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]': finished 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[30312]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[42466]: pgmap v781: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-24T11:51:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]': finished 2024-03-24T11:51:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[42466]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T11:51:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:28.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:28 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:28 smithi119 ceph-mon[33714]: pgmap v781: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:51:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-24T11:51:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-24T11:51:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_count","val": "1"}]': finished 2024-03-24T11:51:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:28 smithi119 ceph-mon[33714]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T11:51:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:28 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:30.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[30312]: pgmap v784: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[30312]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[42466]: pgmap v784: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[42466]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-24T11:51:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:30 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-24T11:51:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:30 smithi119 ceph-mon[33714]: pgmap v784: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:51:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:51:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:30 smithi119 ceph-mon[33714]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T11:51:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-24T11:51:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:30 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-24T11:51:31.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]': finished 2024-03-24T11:51:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[30312]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T11:51:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:31.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]': finished 2024-03-24T11:51:31.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[42466]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T11:51:31.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:31.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:31 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:31 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-98","var": "target_max_objects","val": "250"}]': finished 2024-03-24T11:51:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:31 smithi119 ceph-mon[33714]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T11:51:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:31 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:32.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:51:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:51:32.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:32 smithi119 ceph-mon[33714]: pgmap v787: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:32.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:32 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:51:32.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:32 smithi119 ceph-mon[33714]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T11:51:32.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:32.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:32 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:32.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[42466]: pgmap v787: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:32.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:51:32.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[42466]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T11:51:32.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:32.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:32.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[30312]: pgmap v787: 356 pgs: 356 active+clean; 8.4 MiB data, 672 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:32.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:51:32.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[30312]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T11:51:32.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:32.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:32 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]: dispatch 2024-03-24T11:51:33.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]': finished 2024-03-24T11:51:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[42466]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T11:51:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]': finished 2024-03-24T11:51:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[30312]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T11:51:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:33 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:33 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-98"}]': finished 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:33 smithi119 ceph-mon[33714]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T11:51:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:33 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:34.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:34 smithi114 ceph-mon[30312]: pgmap v790: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:34.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:34 smithi114 ceph-mon[30312]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T11:51:34.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:34 smithi114 ceph-mon[42466]: pgmap v790: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:34 smithi114 ceph-mon[42466]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T11:51:34.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:34 smithi119 ceph-mon[33714]: pgmap v790: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:34 smithi119 ceph-mon[33714]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T11:51:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:51:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:51:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:51:35.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[42466]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T11:51:35.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:35.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:35.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:35.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[42466]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T11:51:35.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:35.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[30312]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T11:51:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[30312]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T11:51:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:35 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:35.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:35 smithi119 ceph-mon[33714]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T11:51:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:35 smithi119 ceph-mon[33714]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T11:51:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:35.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:35 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[30312]: pgmap v793: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[30312]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[42466]: pgmap v793: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[42466]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T11:51:36.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:36.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:36 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:36.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:36 smithi119 ceph-mon[33714]: pgmap v793: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:51:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:51:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:36 smithi119 ceph-mon[33714]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T11:51:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:36.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:36 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]': finished 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]: dispatch 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[30312]: osdmap e515: 8 total, 8 up, 8 in 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]: dispatch 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:51:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]': finished 2024-03-24T11:51:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]: dispatch 2024-03-24T11:51:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[42466]: osdmap e515: 8 total, 8 up, 8 in 2024-03-24T11:51:37.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:37 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]: dispatch 2024-03-24T11:51:37.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:37 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:51:37.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:37 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:51:37.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:37 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:51:37.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-6", "overlaypool": "test-rados-api-smithi114-82677-100"}]': finished 2024-03-24T11:51:37.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]: dispatch 2024-03-24T11:51:37.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:37 smithi119 ceph-mon[33714]: osdmap e515: 8 total, 8 up, 8 in 2024-03-24T11:51:37.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:37 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]: dispatch 2024-03-24T11:51:38.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[30312]: pgmap v796: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]': finished 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[30312]: osdmap e516: 8 total, 8 up, 8 in 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[42466]: pgmap v796: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]': finished 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[42466]: osdmap e516: 8 total, 8 up, 8 in 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:51:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:38 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:51:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:38 smithi119 ceph-mon[33714]: pgmap v796: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:38 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:51:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-100", "mode": "writeback"}]': finished 2024-03-24T11:51:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:38 smithi119 ceph-mon[33714]: osdmap e516: 8 total, 8 up, 8 in 2024-03-24T11:51:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:51:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:38 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[30312]: pgmap v799: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[30312]: osdmap e517: 8 total, 8 up, 8 in 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[42466]: pgmap v799: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:51:40.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[42466]: osdmap e517: 8 total, 8 up, 8 in 2024-03-24T11:51:40.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:40.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:40 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:40.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:40 smithi119 ceph-mon[33714]: pgmap v799: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:40.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:51:40.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:40 smithi119 ceph-mon[33714]: osdmap e517: 8 total, 8 up, 8 in 2024-03-24T11:51:40.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:40.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:40 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:51:41.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:51:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:51:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[42466]: osdmap e518: 8 total, 8 up, 8 in 2024-03-24T11:51:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:51:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:51:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:51:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[30312]: osdmap e518: 8 total, 8 up, 8 in 2024-03-24T11:51:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:41 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:51:41.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:51:41.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:51:41.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:41 smithi119 ceph-mon[33714]: osdmap e518: 8 total, 8 up, 8 in 2024-03-24T11:51:41.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:41 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:51:42.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:51:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:51:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:42 smithi119 ceph-mon[33714]: pgmap v802: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:42 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:51:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:51:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-24T11:51:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:42 smithi119 ceph-mon[33714]: osdmap e519: 8 total, 8 up, 8 in 2024-03-24T11:51:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:42 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[42466]: pgmap v802: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[42466]: osdmap e519: 8 total, 8 up, 8 in 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[30312]: pgmap v802: 356 pgs: 356 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:42.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:51:42.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:51:42.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-24T11:51:42.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[30312]: osdmap e519: 8 total, 8 up, 8 in 2024-03-24T11:51:42.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:42 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-24T11:51:43.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[30312]: osdmap e520: 8 total, 8 up, 8 in 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[42466]: osdmap e520: 8 total, 8 up, 8 in 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:43.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:43 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:43.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:43.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-24T11:51:43.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:43 smithi119 ceph-mon[33714]: osdmap e520: 8 total, 8 up, 8 in 2024-03-24T11:51:43.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:43.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:43 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:44.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[30312]: pgmap v805: 356 pgs: 356 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[30312]: osdmap e521: 8 total, 8 up, 8 in 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[42466]: pgmap v805: 356 pgs: 356 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[42466]: osdmap e521: 8 total, 8 up, 8 in 2024-03-24T11:51:44.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:44 smithi119 ceph-mon[33714]: pgmap v805: 356 pgs: 356 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:44 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]': finished 2024-03-24T11:51:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:44 smithi119 ceph-mon[33714]: osdmap e521: 8 total, 8 up, 8 in 2024-03-24T11:51:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:44 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]: dispatch 2024-03-24T11:51:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:51:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:51:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:51:45.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:45 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]': finished 2024-03-24T11:51:45.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:45 smithi114 ceph-mon[30312]: osdmap e522: 8 total, 8 up, 8 in 2024-03-24T11:51:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:45 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]': finished 2024-03-24T11:51:45.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:45 smithi114 ceph-mon[42466]: osdmap e522: 8 total, 8 up, 8 in 2024-03-24T11:51:45.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:45 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-100"}]': finished 2024-03-24T11:51:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:45 smithi119 ceph-mon[33714]: osdmap e522: 8 total, 8 up, 8 in 2024-03-24T11:51:46.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:46 smithi114 ceph-mon[30312]: pgmap v808: 356 pgs: 356 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:46.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:46 smithi114 ceph-mon[30312]: osdmap e523: 8 total, 8 up, 8 in 2024-03-24T11:51:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:46 smithi114 ceph-mon[42466]: pgmap v808: 356 pgs: 356 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:46 smithi114 ceph-mon[42466]: osdmap e523: 8 total, 8 up, 8 in 2024-03-24T11:51:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:46 smithi119 ceph-mon[33714]: pgmap v808: 356 pgs: 356 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:46 smithi119 ceph-mon[33714]: osdmap e523: 8 total, 8 up, 8 in 2024-03-24T11:51:47.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:47 smithi114 ceph-mon[42466]: osdmap e524: 8 total, 8 up, 8 in 2024-03-24T11:51:47.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:47.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:47 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:47 smithi114 ceph-mon[30312]: osdmap e524: 8 total, 8 up, 8 in 2024-03-24T11:51:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:47.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:47 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:47 smithi119 ceph-mon[33714]: osdmap e524: 8 total, 8 up, 8 in 2024-03-24T11:51:47.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:47.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:47 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[30312]: pgmap v811: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[30312]: osdmap e525: 8 total, 8 up, 8 in 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:51:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[42466]: pgmap v811: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[42466]: osdmap e525: 8 total, 8 up, 8 in 2024-03-24T11:51:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:51:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:51:48.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:48 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:51:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:48 smithi119 ceph-mon[33714]: pgmap v811: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:48 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:48 smithi119 ceph-mon[33714]: osdmap e525: 8 total, 8 up, 8 in 2024-03-24T11:51:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-24T11:51:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:51:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:48 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-24T11:51:49.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[30312]: osdmap e526: 8 total, 8 up, 8 in 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[42466]: osdmap e526: 8 total, 8 up, 8 in 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:51:49.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:49.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:49 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-24T11:51:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:49 smithi119 ceph-mon[33714]: osdmap e526: 8 total, 8 up, 8 in 2024-03-24T11:51:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:51:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:49 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-24T11:51:49.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:51:50.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[30312]: pgmap v814: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[30312]: osdmap e527: 8 total, 8 up, 8 in 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[42466]: pgmap v814: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[42466]: osdmap e527: 8 total, 8 up, 8 in 2024-03-24T11:51:50.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:50 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:51:50.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:50 smithi119 ceph-mon[33714]: pgmap v814: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:51:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:50 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-24T11:51:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:51:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:50 smithi119 ceph-mon[33714]: osdmap e527: 8 total, 8 up, 8 in 2024-03-24T11:51:50.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:50 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-24T11:51:51.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[30312]: osdmap e528: 8 total, 8 up, 8 in 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-102"}]: dispatch 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[30312]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-102"}]: dispatch 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[42466]: osdmap e528: 8 total, 8 up, 8 in 2024-03-24T11:51:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-102"}]: dispatch 2024-03-24T11:51:51.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:51 smithi114 ceph-mon[42466]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-102"}]: dispatch 2024-03-24T11:51:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-24T11:51:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:51 smithi119 ceph-mon[33714]: osdmap e528: 8 total, 8 up, 8 in 2024-03-24T11:51:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-6"}]: dispatch 2024-03-24T11:51:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3563613347' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-102"}]: dispatch 2024-03-24T11:51:51.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:51 smithi119 ceph-mon[33714]: from='client.27358 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-6", "tierpool": "test-rados-api-smithi114-82677-102"}]: dispatch 2024-03-24T11:51:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:52 smithi119 ceph-mon[33714]: pgmap v817: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:52.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:52 smithi119 ceph-mon[33714]: osdmap e529: 8 total, 8 up, 8 in 2024-03-24T11:51:52.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:51:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:51:52.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:52 smithi114 ceph-mon[30312]: pgmap v817: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:52.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:52 smithi114 ceph-mon[30312]: osdmap e529: 8 total, 8 up, 8 in 2024-03-24T11:51:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:52 smithi114 ceph-mon[42466]: pgmap v817: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:52.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:52 smithi114 ceph-mon[42466]: osdmap e529: 8 total, 8 up, 8 in 2024-03-24T11:51:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: osdmap e530: 8 total, 8 up, 8 in 2024-03-24T11:51:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:51:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:51:53.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:53 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: osdmap e530: 8 total, 8 up, 8 in 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:51:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: osdmap e530: 8 total, 8 up, 8 in 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:51:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:53 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:54 smithi119 ceph-mon[33714]: pgmap v820: 292 pgs: 292 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:54 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:51:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:54 smithi119 ceph-mon[33714]: osdmap e531: 8 total, 8 up, 8 in 2024-03-24T11:51:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:54.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:54 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:54.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[30312]: pgmap v820: 292 pgs: 292 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:54.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:51:54.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[30312]: osdmap e531: 8 total, 8 up, 8 in 2024-03-24T11:51:54.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:54.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:54.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[42466]: pgmap v820: 292 pgs: 292 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:54.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:51:54.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[42466]: osdmap e531: 8 total, 8 up, 8 in 2024-03-24T11:51:54.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:54.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:51:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:51:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:51:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:51:55.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:55 smithi119 ceph-mon[33714]: osdmap e532: 8 total, 8 up, 8 in 2024-03-24T11:51:55.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:55 smithi114 ceph-mon[30312]: osdmap e532: 8 total, 8 up, 8 in 2024-03-24T11:51:55.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:55 smithi114 ceph-mon[42466]: osdmap e532: 8 total, 8 up, 8 in 2024-03-24T11:51:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:56 smithi119 ceph-mon[33714]: pgmap v823: 292 pgs: 292 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:56 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:51:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:56 smithi119 ceph-mon[33714]: osdmap e533: 8 total, 8 up, 8 in 2024-03-24T11:51:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:56 smithi114 ceph-mon[30312]: pgmap v823: 292 pgs: 292 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:56 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:51:56.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:56 smithi114 ceph-mon[30312]: osdmap e533: 8 total, 8 up, 8 in 2024-03-24T11:51:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:56 smithi114 ceph-mon[42466]: pgmap v823: 292 pgs: 292 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:51:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:56 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi114-82677-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:51:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:56 smithi114 ceph-mon[42466]: osdmap e533: 8 total, 8 up, 8 in 2024-03-24T11:51:57.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:57 smithi119 ceph-mon[33714]: osdmap e534: 8 total, 8 up, 8 in 2024-03-24T11:51:57.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:57 smithi114 ceph-mon[30312]: osdmap e534: 8 total, 8 up, 8 in 2024-03-24T11:51:57.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:57 smithi114 ceph-mon[42466]: osdmap e534: 8 total, 8 up, 8 in 2024-03-24T11:51:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:58 smithi119 ceph-mon[33714]: pgmap v826: 300 pgs: 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:58 smithi119 ceph-mon[33714]: osdmap e535: 8 total, 8 up, 8 in 2024-03-24T11:51:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:58 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:58 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:58.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[42466]: pgmap v826: 300 pgs: 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[42466]: osdmap e535: 8 total, 8 up, 8 in 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[30312]: pgmap v826: 300 pgs: 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[30312]: osdmap e535: 8 total, 8 up, 8 in 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:51:58.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:58 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:51:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:59 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:51:59 smithi119 ceph-mon[33714]: osdmap e536: 8 total, 8 up, 8 in 2024-03-24T11:51:59.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:59 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:59.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:51:59 smithi114 ceph-mon[30312]: osdmap e536: 8 total, 8 up, 8 in 2024-03-24T11:51:59.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:59 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:51:59.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:51:59 smithi114 ceph-mon[42466]: osdmap e536: 8 total, 8 up, 8 in 2024-03-24T11:52:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:00 smithi119 ceph-mon[33714]: pgmap v829: 332 pgs: 32 unknown, 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:00 smithi119 ceph-mon[33714]: osdmap e537: 8 total, 8 up, 8 in 2024-03-24T11:52:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:00 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:00.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:00 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:00.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[42466]: pgmap v829: 332 pgs: 32 unknown, 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:00.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[42466]: osdmap e537: 8 total, 8 up, 8 in 2024-03-24T11:52:00.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:00.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[30312]: pgmap v829: 332 pgs: 32 unknown, 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[30312]: osdmap e537: 8 total, 8 up, 8 in 2024-03-24T11:52:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:00 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:01.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[30312]: osdmap e538: 8 total, 8 up, 8 in 2024-03-24T11:52:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:01.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[42466]: osdmap e538: 8 total, 8 up, 8 in 2024-03-24T11:52:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:01.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:01 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:01.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:01 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:01 smithi119 ceph-mon[33714]: osdmap e538: 8 total, 8 up, 8 in 2024-03-24T11:52:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:01 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:01.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:01 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:02.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:52:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:52:02.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[42466]: pgmap v832: 364 pgs: 64 unknown, 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:02.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:02.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[42466]: osdmap e539: 8 total, 8 up, 8 in 2024-03-24T11:52:02.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:02.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[30312]: pgmap v832: 364 pgs: 64 unknown, 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[30312]: osdmap e539: 8 total, 8 up, 8 in 2024-03-24T11:52:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:02.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:02 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:02 smithi119 ceph-mon[33714]: pgmap v832: 364 pgs: 64 unknown, 4 creating+activating, 296 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:02 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:02 smithi119 ceph-mon[33714]: osdmap e539: 8 total, 8 up, 8 in 2024-03-24T11:52:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:02 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:02.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:02 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:03.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: osdmap e540: 8 total, 8 up, 8 in 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]': finished 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[42466]: osdmap e541: 8 total, 8 up, 8 in 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: osdmap e540: 8 total, 8 up, 8 in 2024-03-24T11:52:03.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]': finished 2024-03-24T11:52:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:03 smithi114 ceph-mon[30312]: osdmap e541: 8 total, 8 up, 8 in 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-107", "overlaypool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: osdmap e540: 8 total, 8 up, 8 in 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-107-cache", "mode": "writeback"}]': finished 2024-03-24T11:52:03.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:03 smithi119 ceph-mon[33714]: osdmap e541: 8 total, 8 up, 8 in 2024-03-24T11:52:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[30312]: pgmap v835: 364 pgs: 364 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]: dispatch 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]: dispatch 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[42466]: pgmap v835: 364 pgs: 364 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]: dispatch 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]: dispatch 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:04.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:04 smithi119 ceph-mon[33714]: pgmap v835: 364 pgs: 364 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:52:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:04 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]: dispatch 2024-03-24T11:52:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:04 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]: dispatch 2024-03-24T11:52:04.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:04.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:52:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:52:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:52:05.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]': finished 2024-03-24T11:52:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[42466]: osdmap e542: 8 total, 8 up, 8 in 2024-03-24T11:52:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:05.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]': finished 2024-03-24T11:52:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[30312]: osdmap e542: 8 total, 8 up, 8 in 2024-03-24T11:52:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:05.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:05 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:05 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-107"}]': finished 2024-03-24T11:52:05.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:05 smithi119 ceph-mon[33714]: osdmap e542: 8 total, 8 up, 8 in 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/109443587' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:05.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:05 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]: dispatch 2024-03-24T11:52:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[42466]: pgmap v838: 364 pgs: 364 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:52:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[42466]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[42466]: osdmap e543: 8 total, 8 up, 8 in 2024-03-24T11:52:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[30312]: pgmap v838: 364 pgs: 364 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:52:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[30312]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:06.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:06 smithi114 ceph-mon[30312]: osdmap e543: 8 total, 8 up, 8 in 2024-03-24T11:52:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:06 smithi119 ceph-mon[33714]: pgmap v838: 364 pgs: 364 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:52:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:06 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:06 smithi119 ceph-mon[33714]: from='client.56057 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-107", "tierpool": "test-rados-api-smithi114-82677-107-cache"}]': finished 2024-03-24T11:52:06.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:06 smithi119 ceph-mon[33714]: osdmap e543: 8 total, 8 up, 8 in 2024-03-24T11:52:07.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:07 smithi114 ceph-mon[30312]: osdmap e544: 8 total, 8 up, 8 in 2024-03-24T11:52:07.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:07 smithi114 ceph-mon[30312]: osdmap e545: 8 total, 8 up, 8 in 2024-03-24T11:52:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:07 smithi114 ceph-mon[42466]: osdmap e544: 8 total, 8 up, 8 in 2024-03-24T11:52:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:07 smithi114 ceph-mon[42466]: osdmap e545: 8 total, 8 up, 8 in 2024-03-24T11:52:07.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:07 smithi119 ceph-mon[33714]: osdmap e544: 8 total, 8 up, 8 in 2024-03-24T11:52:07.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:07 smithi119 ceph-mon[33714]: osdmap e545: 8 total, 8 up, 8 in 2024-03-24T11:52:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[30312]: pgmap v841: 332 pgs: 332 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:08.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[42466]: pgmap v841: 332 pgs: 332 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:08 smithi119 ceph-mon[33714]: pgmap v841: 332 pgs: 332 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:09.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:52:09.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:09 smithi114 ceph-mon[30312]: osdmap e546: 8 total, 8 up, 8 in 2024-03-24T11:52:09.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:52:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:09 smithi114 ceph-mon[42466]: osdmap e546: 8 total, 8 up, 8 in 2024-03-24T11:52:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:52:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:09 smithi119 ceph-mon[33714]: osdmap e546: 8 total, 8 up, 8 in 2024-03-24T11:52:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:10 smithi114 ceph-mon[30312]: pgmap v844: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:10 smithi114 ceph-mon[30312]: osdmap e547: 8 total, 8 up, 8 in 2024-03-24T11:52:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:10 smithi114 ceph-mon[42466]: pgmap v844: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:10.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:10 smithi114 ceph-mon[42466]: osdmap e547: 8 total, 8 up, 8 in 2024-03-24T11:52:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:10 smithi119 ceph-mon[33714]: pgmap v844: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:10 smithi119 ceph-mon[33714]: osdmap e547: 8 total, 8 up, 8 in 2024-03-24T11:52:11.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:11.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:11 smithi114 ceph-mon[42466]: osdmap e548: 8 total, 8 up, 8 in 2024-03-24T11:52:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:11 smithi114 ceph-mon[42466]: osdmap e549: 8 total, 8 up, 8 in 2024-03-24T11:52:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:11 smithi114 ceph-mon[30312]: osdmap e548: 8 total, 8 up, 8 in 2024-03-24T11:52:11.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:11 smithi114 ceph-mon[30312]: osdmap e549: 8 total, 8 up, 8 in 2024-03-24T11:52:11.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:11 smithi119 ceph-mon[33714]: osdmap e548: 8 total, 8 up, 8 in 2024-03-24T11:52:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:11 smithi119 ceph-mon[33714]: osdmap e549: 8 total, 8 up, 8 in 2024-03-24T11:52:12.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:52:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[30312]: pgmap v847: 308 pgs: 8 unknown, 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[30312]: osdmap e550: 8 total, 8 up, 8 in 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[42466]: pgmap v847: 308 pgs: 8 unknown, 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[42466]: osdmap e550: 8 total, 8 up, 8 in 2024-03-24T11:52:12.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:12.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:12 smithi119 ceph-mon[33714]: pgmap v847: 308 pgs: 8 unknown, 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:12 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:12 smithi119 ceph-mon[33714]: osdmap e550: 8 total, 8 up, 8 in 2024-03-24T11:52:12.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:13.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:13.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:13.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[30312]: osdmap e551: 8 total, 8 up, 8 in 2024-03-24T11:52:13.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-109", "overlaypool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[42466]: osdmap e551: 8 total, 8 up, 8 in 2024-03-24T11:52:13.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-109", "overlaypool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:13.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:13.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:13.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:13 smithi119 ceph-mon[33714]: osdmap e551: 8 total, 8 up, 8 in 2024-03-24T11:52:13.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-109", "overlaypool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:14.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[42466]: pgmap v850: 340 pgs: 32 creating+peering, 308 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:14.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-109", "overlaypool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:14.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[42466]: osdmap e552: 8 total, 8 up, 8 in 2024-03-24T11:52:14.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-109-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:14.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[30312]: pgmap v850: 340 pgs: 32 creating+peering, 308 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:14.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-109", "overlaypool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:14.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[30312]: osdmap e552: 8 total, 8 up, 8 in 2024-03-24T11:52:14.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-109-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:14 smithi119 ceph-mon[33714]: pgmap v850: 340 pgs: 32 creating+peering, 308 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-109", "overlaypool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:14 smithi119 ceph-mon[33714]: osdmap e552: 8 total, 8 up, 8 in 2024-03-24T11:52:14.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-109-cache", "mode": "writeback"}]: dispatch 2024-03-24T11:52:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:52:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:52:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-109-cache", "mode": "writeback"}]': finished 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[42466]: osdmap e553: 8 total, 8 up, 8 in 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-109-cache", "mode": "writeback"}]': finished 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[30312]: osdmap e553: 8 total, 8 up, 8 in 2024-03-24T11:52:15.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:52:15.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:15 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-109-cache", "mode": "writeback"}]': finished 2024-03-24T11:52:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:15 smithi119 ceph-mon[33714]: osdmap e553: 8 total, 8 up, 8 in 2024-03-24T11:52:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:52:16.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[30312]: pgmap v853: 340 pgs: 32 creating+peering, 308 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:16.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:52:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[30312]: osdmap e554: 8 total, 8 up, 8 in 2024-03-24T11:52:16.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:52:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[42466]: pgmap v853: 340 pgs: 32 creating+peering, 308 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:52:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[42466]: osdmap e554: 8 total, 8 up, 8 in 2024-03-24T11:52:16.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:52:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:16 smithi119 ceph-mon[33714]: pgmap v853: 340 pgs: 32 creating+peering, 308 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:52:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:16 smithi119 ceph-mon[33714]: osdmap e554: 8 total, 8 up, 8 in 2024-03-24T11:52:16.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:52:18.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[42466]: pgmap v856: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:18.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:52:18.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[42466]: osdmap e555: 8 total, 8 up, 8 in 2024-03-24T11:52:18.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:52:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[30312]: pgmap v856: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:52:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[30312]: osdmap e555: 8 total, 8 up, 8 in 2024-03-24T11:52:18.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:52:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:18 smithi119 ceph-mon[33714]: pgmap v856: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:52:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:18 smithi119 ceph-mon[33714]: osdmap e555: 8 total, 8 up, 8 in 2024-03-24T11:52:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:52:19.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: osdmap e556: 8 total, 8 up, 8 in 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[30312]: osdmap e557: 8 total, 8 up, 8 in 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:19.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:52:19.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: osdmap e556: 8 total, 8 up, 8 in 2024-03-24T11:52:19.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-24T11:52:19.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:19.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:19.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-24T11:52:19.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:19 smithi114 ceph-mon[42466]: osdmap e557: 8 total, 8 up, 8 in 2024-03-24T11:52:19.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:52:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: osdmap e556: 8 total, 8 up, 8 in 2024-03-24T11:52:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-24T11:52:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-24T11:52:19.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:19 smithi119 ceph-mon[33714]: osdmap e557: 8 total, 8 up, 8 in 2024-03-24T11:52:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:20 smithi114 ceph-mon[30312]: pgmap v859: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:20.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:20.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:20 smithi114 ceph-mon[42466]: pgmap v859: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:20.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:20 smithi119 ceph-mon[33714]: pgmap v859: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:21.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:21.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:21 smithi114 ceph-mon[42466]: osdmap e558: 8 total, 8 up, 8 in 2024-03-24T11:52:21.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:21 smithi114 ceph-mon[30312]: osdmap e558: 8 total, 8 up, 8 in 2024-03-24T11:52:21.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:21.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:21 smithi119 ceph-mon[33714]: osdmap e558: 8 total, 8 up, 8 in 2024-03-24T11:52:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]: dispatch 2024-03-24T11:52:22.419 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:52:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:52:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:22 smithi114 ceph-mon[42466]: pgmap v862: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:22 smithi114 ceph-mon[42466]: osdmap e559: 8 total, 8 up, 8 in 2024-03-24T11:52:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:22 smithi114 ceph-mon[30312]: pgmap v862: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:22.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:22 smithi114 ceph-mon[30312]: osdmap e559: 8 total, 8 up, 8 in 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:22 smithi119 ceph-mon[33714]: pgmap v862: 340 pgs: 340 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:22.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-109", "tierpool": "test-rados-api-smithi114-82677-109-cache"}]': finished 2024-03-24T11:52:22.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:22 smithi119 ceph-mon[33714]: osdmap e559: 8 total, 8 up, 8 in 2024-03-24T11:52:23.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:23.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:23 smithi114 ceph-mon[30312]: osdmap e560: 8 total, 8 up, 8 in 2024-03-24T11:52:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:23.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:23 smithi114 ceph-mon[42466]: osdmap e560: 8 total, 8 up, 8 in 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:23.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:23 smithi119 ceph-mon[33714]: osdmap e560: 8 total, 8 up, 8 in 2024-03-24T11:52:24.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:24 smithi114 ceph-mon[30312]: pgmap v865: 308 pgs: 308 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:24.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:24 smithi114 ceph-mon[30312]: osdmap e561: 8 total, 8 up, 8 in 2024-03-24T11:52:24.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:24.739 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:52:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:52:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:52:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:24 smithi114 ceph-mon[42466]: pgmap v865: 308 pgs: 308 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:24 smithi114 ceph-mon[42466]: osdmap e561: 8 total, 8 up, 8 in 2024-03-24T11:52:24.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:24 smithi119 ceph-mon[33714]: pgmap v865: 308 pgs: 308 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:24 smithi119 ceph-mon[33714]: osdmap e561: 8 total, 8 up, 8 in 2024-03-24T11:52:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:25.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:25.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[42466]: osdmap e562: 8 total, 8 up, 8 in 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[42466]: osdmap e563: 8 total, 8 up, 8 in 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[30312]: osdmap e562: 8 total, 8 up, 8 in 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:25.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:25 smithi114 ceph-mon[30312]: osdmap e563: 8 total, 8 up, 8 in 2024-03-24T11:52:25.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:25.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:25 smithi119 ceph-mon[33714]: osdmap e562: 8 total, 8 up, 8 in 2024-03-24T11:52:25.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]: dispatch 2024-03-24T11:52:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1221393574' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-109"}]': finished 2024-03-24T11:52:25.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:25 smithi119 ceph-mon[33714]: osdmap e563: 8 total, 8 up, 8 in 2024-03-24T11:52:26.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:26 smithi114 ceph-mon[30312]: pgmap v868: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:26.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:26 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:26.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:26 smithi114 ceph-mon[42466]: pgmap v868: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:26.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:26 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:26.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:26 smithi119 ceph-mon[33714]: pgmap v868: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:26.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:26 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:27.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[42466]: osdmap e564: 8 total, 8 up, 8 in 2024-03-24T11:52:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:27.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:52:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[30312]: osdmap e564: 8 total, 8 up, 8 in 2024-03-24T11:52:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:27.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:27 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:52:27.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:27 smithi119 ceph-mon[33714]: osdmap e564: 8 total, 8 up, 8 in 2024-03-24T11:52:27.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:27 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:27.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:27 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[30312]: pgmap v871: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[30312]: osdmap e565: 8 total, 8 up, 8 in 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[30312]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[30312]: osdmap e566: 8 total, 8 up, 8 in 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[42466]: pgmap v871: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[42466]: osdmap e565: 8 total, 8 up, 8 in 2024-03-24T11:52:28.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[42466]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:52:28.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:28 smithi114 ceph-mon[42466]: osdmap e566: 8 total, 8 up, 8 in 2024-03-24T11:52:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:28 smithi119 ceph-mon[33714]: pgmap v871: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T11:52:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:28 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/798446395' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:28 smithi119 ceph-mon[33714]: osdmap e565: 8 total, 8 up, 8 in 2024-03-24T11:52:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:28 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]: dispatch 2024-03-24T11:52:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:28 smithi119 ceph-mon[33714]: from='client.49078 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi114-82677-104"}]': finished 2024-03-24T11:52:28.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:28 smithi119 ceph-mon[33714]: osdmap e566: 8 total, 8 up, 8 in 2024-03-24T11:52:29.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:29.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:29 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:29.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:29 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:29 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:29.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:29 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T11:52:30.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[30312]: pgmap v874: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[30312]: osdmap e567: 8 total, 8 up, 8 in 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[42466]: pgmap v874: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[42466]: osdmap e567: 8 total, 8 up, 8 in 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:30.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:30 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:30 smithi119 ceph-mon[33714]: pgmap v874: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T11:52:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:30 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi114-82677-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-24T11:52:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:30 smithi119 ceph-mon[33714]: osdmap e567: 8 total, 8 up, 8 in 2024-03-24T11:52:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:31.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:30 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:31.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:31 smithi114 ceph-mon[30312]: osdmap e568: 8 total, 8 up, 8 in 2024-03-24T11:52:31.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:31.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:31 smithi114 ceph-mon[30312]: osdmap e569: 8 total, 8 up, 8 in 2024-03-24T11:52:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:31 smithi114 ceph-mon[42466]: osdmap e568: 8 total, 8 up, 8 in 2024-03-24T11:52:31.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:31.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:31 smithi114 ceph-mon[42466]: osdmap e569: 8 total, 8 up, 8 in 2024-03-24T11:52:32.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:31 smithi119 ceph-mon[33714]: osdmap e568: 8 total, 8 up, 8 in 2024-03-24T11:52:32.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi114-82677-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:32.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:31 smithi119 ceph-mon[33714]: osdmap e569: 8 total, 8 up, 8 in 2024-03-24T11:52:32.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:52:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:52:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[30312]: pgmap v877: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[30312]: osdmap e570: 8 total, 8 up, 8 in 2024-03-24T11:52:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:32.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[42466]: pgmap v877: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[42466]: osdmap e570: 8 total, 8 up, 8 in 2024-03-24T11:52:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:32.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:32 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:33.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:32 smithi119 ceph-mon[33714]: pgmap v877: 292 pgs: 292 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:33.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:32 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:33.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:32 smithi119 ceph-mon[33714]: osdmap e570: 8 total, 8 up, 8 in 2024-03-24T11:52:33.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:33.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:32 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:33.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:33.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[30312]: pgmap v880: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[30312]: osdmap e571: 8 total, 8 up, 8 in 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[42466]: pgmap v880: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[42466]: osdmap e571: 8 total, 8 up, 8 in 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:33.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:33 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:34.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:33 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:34.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:33 smithi119 ceph-mon[33714]: pgmap v880: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:34.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:33 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:34.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:33 smithi119 ceph-mon[33714]: osdmap e571: 8 total, 8 up, 8 in 2024-03-24T11:52:34.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:34.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:33 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[42466]: osdmap e572: 8 total, 8 up, 8 in 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[30312]: osdmap e572: 8 total, 8 up, 8 in 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:34.990 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:52:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:52:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:52:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:34 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:34 smithi119 ceph-mon[33714]: osdmap e572: 8 total, 8 up, 8 in 2024-03-24T11:52:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:34 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[30312]: pgmap v883: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]': finished 2024-03-24T11:52:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[30312]: osdmap e573: 8 total, 8 up, 8 in 2024-03-24T11:52:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:36.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[42466]: pgmap v883: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:36.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]': finished 2024-03-24T11:52:36.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[42466]: osdmap e573: 8 total, 8 up, 8 in 2024-03-24T11:52:36.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:36.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:36 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:36 smithi119 ceph-mon[33714]: pgmap v883: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:36 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-112"}]': finished 2024-03-24T11:52:36.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:36 smithi119 ceph-mon[33714]: osdmap e573: 8 total, 8 up, 8 in 2024-03-24T11:52:36.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:36 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:36.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:36 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:37.552 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:37 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:37.552 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:37 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:37.552 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:37 smithi119 ceph-mon[33714]: osdmap e574: 8 total, 8 up, 8 in 2024-03-24T11:52:37.552 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:37.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:37 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:37.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[42466]: osdmap e574: 8 total, 8 up, 8 in 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[30312]: osdmap e574: 8 total, 8 up, 8 in 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:37.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:37 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]: dispatch 2024-03-24T11:52:38.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[30312]: pgmap v886: 332 pgs: 332 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:52:38.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]': finished 2024-03-24T11:52:38.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[30312]: osdmap e575: 8 total, 8 up, 8 in 2024-03-24T11:52:38.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[42466]: pgmap v886: 332 pgs: 332 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:52:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]': finished 2024-03-24T11:52:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[42466]: osdmap e575: 8 total, 8 up, 8 in 2024-03-24T11:52:38.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:38 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:38 smithi119 ceph-mon[33714]: pgmap v886: 332 pgs: 332 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:52:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:38 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-112"}]': finished 2024-03-24T11:52:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:38 smithi119 ceph-mon[33714]: osdmap e575: 8 total, 8 up, 8 in 2024-03-24T11:52:38.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:38 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:52:39.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:39 smithi114 ceph-mon[30312]: osdmap e576: 8 total, 8 up, 8 in 2024-03-24T11:52:39.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:39 smithi114 ceph-mon[42466]: osdmap e576: 8 total, 8 up, 8 in 2024-03-24T11:52:39.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:39 smithi119 ceph-mon[33714]: osdmap e576: 8 total, 8 up, 8 in 2024-03-24T11:52:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:40 smithi119 ceph-mon[33714]: pgmap v889: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:40 smithi119 ceph-mon[33714]: osdmap e577: 8 total, 8 up, 8 in 2024-03-24T11:52:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:40 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:40 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:40.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:40 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[30312]: pgmap v889: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[30312]: osdmap e577: 8 total, 8 up, 8 in 2024-03-24T11:52:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:40.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[42466]: pgmap v889: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[42466]: osdmap e577: 8 total, 8 up, 8 in 2024-03-24T11:52:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:40.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:40 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:41.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:41 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:41.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:41 smithi119 ceph-mon[33714]: osdmap e578: 8 total, 8 up, 8 in 2024-03-24T11:52:41.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:41.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:41 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:41.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:41 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:41.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:41 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:41.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:41.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[30312]: osdmap e578: 8 total, 8 up, 8 in 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[42466]: osdmap e578: 8 total, 8 up, 8 in 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:41.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:41 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:42.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:52:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: pgmap v892: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: osdmap e579: 8 total, 8 up, 8 in 2024-03-24T11:52:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:42.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:42 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: pgmap v892: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: osdmap e579: 8 total, 8 up, 8 in 2024-03-24T11:52:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: pgmap v892: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: osdmap e579: 8 total, 8 up, 8 in 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:52:42.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:42 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:52:43.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:43 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]': finished 2024-03-24T11:52:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:43 smithi119 ceph-mon[33714]: osdmap e580: 8 total, 8 up, 8 in 2024-03-24T11:52:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]: dispatch 2024-03-24T11:52:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:43 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]: dispatch 2024-03-24T11:52:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:43 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:43.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:43 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]': finished 2024-03-24T11:52:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]': finished 2024-03-24T11:52:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[30312]: osdmap e580: 8 total, 8 up, 8 in 2024-03-24T11:52:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]: dispatch 2024-03-24T11:52:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]: dispatch 2024-03-24T11:52:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]': finished 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-114"}]': finished 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[42466]: osdmap e580: 8 total, 8 up, 8 in 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]: dispatch 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]: dispatch 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:43.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:43 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-114", "mode": "writeback"}]': finished 2024-03-24T11:52:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:44 smithi119 ceph-mon[33714]: pgmap v895: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:44 smithi119 ceph-mon[33714]: osdmap e581: 8 total, 8 up, 8 in 2024-03-24T11:52:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:44.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:44 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[30312]: pgmap v895: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[30312]: osdmap e581: 8 total, 8 up, 8 in 2024-03-24T11:52:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[42466]: pgmap v895: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[42466]: osdmap e581: 8 total, 8 up, 8 in 2024-03-24T11:52:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:52:44.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:52:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:52:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:52:45.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:45.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:45 smithi119 ceph-mon[33714]: osdmap e582: 8 total, 8 up, 8 in 2024-03-24T11:52:45.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:45.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:45.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:45.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[30312]: osdmap e582: 8 total, 8 up, 8 in 2024-03-24T11:52:45.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:45.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:45.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:52:45.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[42466]: osdmap e582: 8 total, 8 up, 8 in 2024-03-24T11:52:45.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:45.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]: dispatch 2024-03-24T11:52:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[42466]: pgmap v898: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]': finished 2024-03-24T11:52:46.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[42466]: osdmap e583: 8 total, 8 up, 8 in 2024-03-24T11:52:46.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[30312]: pgmap v898: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:46.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:46.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]': finished 2024-03-24T11:52:46.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:46 smithi114 ceph-mon[30312]: osdmap e583: 8 total, 8 up, 8 in 2024-03-24T11:52:47.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:46 smithi119 ceph-mon[33714]: pgmap v898: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:52:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:46 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:52:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:46 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-114"}]': finished 2024-03-24T11:52:47.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:46 smithi119 ceph-mon[33714]: osdmap e583: 8 total, 8 up, 8 in 2024-03-24T11:52:47.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:47 smithi114 ceph-mon[30312]: osdmap e584: 8 total, 8 up, 8 in 2024-03-24T11:52:47.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:47 smithi114 ceph-mon[42466]: osdmap e584: 8 total, 8 up, 8 in 2024-03-24T11:52:48.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:47 smithi119 ceph-mon[33714]: osdmap e584: 8 total, 8 up, 8 in 2024-03-24T11:52:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[42466]: pgmap v901: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[42466]: osdmap e585: 8 total, 8 up, 8 in 2024-03-24T11:52:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:48.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[30312]: pgmap v901: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:48.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[30312]: osdmap e585: 8 total, 8 up, 8 in 2024-03-24T11:52:48.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:48.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:49.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:48 smithi119 ceph-mon[33714]: pgmap v901: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:48 smithi119 ceph-mon[33714]: osdmap e585: 8 total, 8 up, 8 in 2024-03-24T11:52:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:49.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:52:49.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:49.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[30312]: osdmap e586: 8 total, 8 up, 8 in 2024-03-24T11:52:49.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:49.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[30312]: pgmap v904: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:49.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:49.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[42466]: osdmap e586: 8 total, 8 up, 8 in 2024-03-24T11:52:49.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:49.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:49 smithi114 ceph-mon[42466]: pgmap v904: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:50.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:49 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:52:50.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:49 smithi119 ceph-mon[33714]: osdmap e586: 8 total, 8 up, 8 in 2024-03-24T11:52:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:52:50.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:49 smithi119 ceph-mon[33714]: pgmap v904: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:52:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:50 smithi114 ceph-mon[30312]: osdmap e587: 8 total, 8 up, 8 in 2024-03-24T11:52:50.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:50.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:50 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:50.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:50 smithi114 ceph-mon[42466]: osdmap e587: 8 total, 8 up, 8 in 2024-03-24T11:52:50.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:50.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:50 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:50 smithi119 ceph-mon[33714]: osdmap e587: 8 total, 8 up, 8 in 2024-03-24T11:52:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:50 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:52:51.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:51.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[30312]: osdmap e588: 8 total, 8 up, 8 in 2024-03-24T11:52:51.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:52:51.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:52:51.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[30312]: pgmap v907: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:51.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:51.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[42466]: osdmap e588: 8 total, 8 up, 8 in 2024-03-24T11:52:51.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:52:51.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:52:51.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:51 smithi114 ceph-mon[42466]: pgmap v907: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:52.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:51 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:52:52.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:51 smithi119 ceph-mon[33714]: osdmap e588: 8 total, 8 up, 8 in 2024-03-24T11:52:52.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:52:52.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:51 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:52:52.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:51 smithi119 ceph-mon[33714]: pgmap v907: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:52:52.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:52:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:52:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]': finished 2024-03-24T11:52:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[30312]: osdmap e589: 8 total, 8 up, 8 in 2024-03-24T11:52:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]: dispatch 2024-03-24T11:52:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]: dispatch 2024-03-24T11:52:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:52.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]': finished 2024-03-24T11:52:52.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[42466]: osdmap e589: 8 total, 8 up, 8 in 2024-03-24T11:52:52.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]: dispatch 2024-03-24T11:52:52.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]: dispatch 2024-03-24T11:52:52.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:52 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:52 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-116"}]': finished 2024-03-24T11:52:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:52 smithi119 ceph-mon[33714]: osdmap e589: 8 total, 8 up, 8 in 2024-03-24T11:52:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]: dispatch 2024-03-24T11:52:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:52 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]: dispatch 2024-03-24T11:52:53.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:52 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:52:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]': finished 2024-03-24T11:52:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[30312]: osdmap e590: 8 total, 8 up, 8 in 2024-03-24T11:52:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "296.6"}]: dispatch 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[30312]: from='mon.? v1:172.21.15.114:0/2564497970' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "296.6"}]: dispatch 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[30312]: pgmap v910: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]': finished 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[42466]: osdmap e590: 8 total, 8 up, 8 in 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "296.6"}]: dispatch 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[42466]: from='mon.? v1:172.21.15.114:0/2564497970' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "296.6"}]: dispatch 2024-03-24T11:52:53.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:53 smithi114 ceph-mon[42466]: pgmap v910: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:52:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:53 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:52:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-116", "mode": "writeback"}]': finished 2024-03-24T11:52:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:53 smithi119 ceph-mon[33714]: osdmap e590: 8 total, 8 up, 8 in 2024-03-24T11:52:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "296.6"}]: dispatch 2024-03-24T11:52:54.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:53 smithi119 ceph-mon[33714]: from='mon.? v1:172.21.15.114:0/2564497970' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "296.6"}]: dispatch 2024-03-24T11:52:54.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:53 smithi119 ceph-mon[33714]: pgmap v910: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:52:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:54 smithi114 ceph-mon[30312]: 296.6 deep-scrub starts 2024-03-24T11:52:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:54 smithi114 ceph-mon[30312]: 296.6 deep-scrub ok 2024-03-24T11:52:54.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:54 smithi114 ceph-mon[42466]: 296.6 deep-scrub starts 2024-03-24T11:52:54.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:54 smithi114 ceph-mon[42466]: 296.6 deep-scrub ok 2024-03-24T11:52:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:52:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:52:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:52:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:54 smithi119 ceph-mon[33714]: 296.6 deep-scrub starts 2024-03-24T11:52:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:54 smithi119 ceph-mon[33714]: 296.6 deep-scrub ok 2024-03-24T11:52:55.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:55 smithi114 ceph-mon[30312]: pgmap v911: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-24T11:52:55.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:55 smithi114 ceph-mon[42466]: pgmap v911: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-24T11:52:56.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:55 smithi119 ceph-mon[33714]: pgmap v911: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-24T11:52:58.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:52:57 smithi119 ceph-mon[33714]: pgmap v912: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-24T11:52:58.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:57 smithi114 ceph-mon[30312]: pgmap v912: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-24T11:52:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:52:57 smithi114 ceph-mon[42466]: pgmap v912: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-24T11:53:00.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:00 smithi119 ceph-mon[33714]: pgmap v913: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-24T11:53:00.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:52:59 smithi114 ceph-mon[30312]: pgmap v913: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-24T11:53:00.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:00 smithi114 ceph-mon[42466]: pgmap v913: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-24T11:53:02.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:02 smithi114 ceph-mon[30312]: pgmap v914: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:53:02.290 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:02 smithi114 ceph-mon[42466]: pgmap v914: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:53:02.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:02 smithi119 ceph-mon[33714]: pgmap v914: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:53:02.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:53:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:53:03.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:03.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:03.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:04.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:04 smithi119 ceph-mon[33714]: pgmap v915: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:04.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:04.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:04 smithi114 ceph-mon[30312]: pgmap v915: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:04 smithi114 ceph-mon[42466]: pgmap v915: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:53:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:53:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:06.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:06 smithi119 ceph-mon[33714]: pgmap v916: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:53:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:06 smithi114 ceph-mon[30312]: pgmap v916: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:53:06.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:06 smithi114 ceph-mon[42466]: pgmap v916: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:53:08.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:08 smithi119 ceph-mon[33714]: pgmap v917: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:53:08.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:08 smithi114 ceph-mon[30312]: pgmap v917: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:53:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:08 smithi114 ceph-mon[42466]: pgmap v917: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:53:09.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:09 smithi119 ceph-mon[33714]: osdmap e591: 8 total, 8 up, 8 in 2024-03-24T11:53:09.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:09.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:09 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:09.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:09 smithi114 ceph-mon[30312]: osdmap e591: 8 total, 8 up, 8 in 2024-03-24T11:53:09.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:09.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:09 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:09 smithi114 ceph-mon[42466]: osdmap e591: 8 total, 8 up, 8 in 2024-03-24T11:53:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:09 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:10.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:10 smithi119 ceph-mon[33714]: pgmap v919: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:10.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:10 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:10 smithi119 ceph-mon[33714]: osdmap e592: 8 total, 8 up, 8 in 2024-03-24T11:53:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:53:10.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:10 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:53:10.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[30312]: pgmap v919: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[30312]: osdmap e592: 8 total, 8 up, 8 in 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[42466]: pgmap v919: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[42466]: osdmap e592: 8 total, 8 up, 8 in 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:53:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:10 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]: dispatch 2024-03-24T11:53:11.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:11 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:11.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:11 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]': finished 2024-03-24T11:53:11.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:11 smithi119 ceph-mon[33714]: osdmap e593: 8 total, 8 up, 8 in 2024-03-24T11:53:11.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:11 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:11.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:11 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]': finished 2024-03-24T11:53:11.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:11 smithi114 ceph-mon[30312]: osdmap e593: 8 total, 8 up, 8 in 2024-03-24T11:53:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:11 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:11 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-116"}]': finished 2024-03-24T11:53:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:11 smithi114 ceph-mon[42466]: osdmap e593: 8 total, 8 up, 8 in 2024-03-24T11:53:12.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:12 smithi119 ceph-mon[33714]: pgmap v922: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:53:12.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:12 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:12.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:12 smithi119 ceph-mon[33714]: osdmap e594: 8 total, 8 up, 8 in 2024-03-24T11:53:12.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:53:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:53:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:12 smithi114 ceph-mon[42466]: pgmap v922: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:53:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:12 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:12 smithi114 ceph-mon[42466]: osdmap e594: 8 total, 8 up, 8 in 2024-03-24T11:53:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:12 smithi114 ceph-mon[30312]: pgmap v922: 332 pgs: 332 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:53:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:12 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:12.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:12 smithi114 ceph-mon[30312]: osdmap e594: 8 total, 8 up, 8 in 2024-03-24T11:53:13.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:13 smithi119 ceph-mon[33714]: osdmap e595: 8 total, 8 up, 8 in 2024-03-24T11:53:13.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:13.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:13 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:13.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:13.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[30312]: osdmap e595: 8 total, 8 up, 8 in 2024-03-24T11:53:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[42466]: osdmap e595: 8 total, 8 up, 8 in 2024-03-24T11:53:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:14 smithi114 ceph-mon[42466]: pgmap v925: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-24T11:53:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:14 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:14 smithi114 ceph-mon[42466]: osdmap e596: 8 total, 8 up, 8 in 2024-03-24T11:53:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:14 smithi114 ceph-mon[30312]: pgmap v925: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-24T11:53:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:14 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:14 smithi114 ceph-mon[30312]: osdmap e596: 8 total, 8 up, 8 in 2024-03-24T11:53:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:14 smithi119 ceph-mon[33714]: pgmap v925: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-24T11:53:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:14 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:14 smithi119 ceph-mon[33714]: osdmap e596: 8 total, 8 up, 8 in 2024-03-24T11:53:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:53:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:53:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:15.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:15 smithi114 ceph-mon[42466]: osdmap e597: 8 total, 8 up, 8 in 2024-03-24T11:53:15.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:15 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:15.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:15 smithi114 ceph-mon[30312]: osdmap e597: 8 total, 8 up, 8 in 2024-03-24T11:53:15.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:15.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:15 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:15 smithi119 ceph-mon[33714]: osdmap e597: 8 total, 8 up, 8 in 2024-03-24T11:53:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:15 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:16.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[30312]: pgmap v928: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-24T11:53:16.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:16.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[30312]: osdmap e598: 8 total, 8 up, 8 in 2024-03-24T11:53:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[42466]: pgmap v928: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-24T11:53:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[42466]: osdmap e598: 8 total, 8 up, 8 in 2024-03-24T11:53:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:16.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:16 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:16 smithi119 ceph-mon[33714]: pgmap v928: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-24T11:53:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:16 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:16 smithi119 ceph-mon[33714]: osdmap e598: 8 total, 8 up, 8 in 2024-03-24T11:53:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:16 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:17.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]': finished 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[42466]: osdmap e599: 8 total, 8 up, 8 in 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]: dispatch 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]: dispatch 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]': finished 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[42466]: osdmap e600: 8 total, 8 up, 8 in 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]': finished 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[30312]: osdmap e599: 8 total, 8 up, 8 in 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]: dispatch 2024-03-24T11:53:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]: dispatch 2024-03-24T11:53:17.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:17.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]': finished 2024-03-24T11:53:17.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:17 smithi114 ceph-mon[30312]: osdmap e600: 8 total, 8 up, 8 in 2024-03-24T11:53:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-118"}]': finished 2024-03-24T11:53:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:17 smithi119 ceph-mon[33714]: osdmap e599: 8 total, 8 up, 8 in 2024-03-24T11:53:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]: dispatch 2024-03-24T11:53:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]: dispatch 2024-03-24T11:53:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:17 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-118", "mode": "writeback"}]': finished 2024-03-24T11:53:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:17 smithi119 ceph-mon[33714]: osdmap e600: 8 total, 8 up, 8 in 2024-03-24T11:53:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[30312]: pgmap v931: 332 pgs: 332 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:53:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[30312]: osdmap e601: 8 total, 8 up, 8 in 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[42466]: pgmap v931: 332 pgs: 332 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[42466]: osdmap e601: 8 total, 8 up, 8 in 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:18 smithi119 ceph-mon[33714]: pgmap v931: 332 pgs: 332 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:53:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:18 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:18 smithi119 ceph-mon[33714]: osdmap e601: 8 total, 8 up, 8 in 2024-03-24T11:53:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:19.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:19.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[30312]: osdmap e602: 8 total, 8 up, 8 in 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[42466]: osdmap e602: 8 total, 8 up, 8 in 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:19 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:19.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:19 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:19 smithi119 ceph-mon[33714]: osdmap e602: 8 total, 8 up, 8 in 2024-03-24T11:53:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:19.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:19 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]: dispatch 2024-03-24T11:53:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[30312]: pgmap v935: 332 pgs: 332 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 529 B/s wr, 1 op/s 2024-03-24T11:53:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]': finished 2024-03-24T11:53:20.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[30312]: osdmap e603: 8 total, 8 up, 8 in 2024-03-24T11:53:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[42466]: pgmap v935: 332 pgs: 332 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 529 B/s wr, 1 op/s 2024-03-24T11:53:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]': finished 2024-03-24T11:53:20.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:20 smithi114 ceph-mon[42466]: osdmap e603: 8 total, 8 up, 8 in 2024-03-24T11:53:20.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:20 smithi119 ceph-mon[33714]: pgmap v935: 332 pgs: 332 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 529 B/s wr, 1 op/s 2024-03-24T11:53:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:20 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:20 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-118"}]': finished 2024-03-24T11:53:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:20 smithi119 ceph-mon[33714]: osdmap e603: 8 total, 8 up, 8 in 2024-03-24T11:53:22.110 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:21 smithi119 ceph-mon[33714]: osdmap e604: 8 total, 8 up, 8 in 2024-03-24T11:53:22.110 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:21 smithi119 ceph-mon[33714]: pgmap v938: 300 pgs: 300 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:22.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:21 smithi114 ceph-mon[42466]: osdmap e604: 8 total, 8 up, 8 in 2024-03-24T11:53:22.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:21 smithi114 ceph-mon[42466]: pgmap v938: 300 pgs: 300 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:21 smithi114 ceph-mon[30312]: osdmap e604: 8 total, 8 up, 8 in 2024-03-24T11:53:22.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:21 smithi114 ceph-mon[30312]: pgmap v938: 300 pgs: 300 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:22.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:53:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:53:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[42466]: osdmap e605: 8 total, 8 up, 8 in 2024-03-24T11:53:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:23.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[30312]: osdmap e605: 8 total, 8 up, 8 in 2024-03-24T11:53:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:23.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:22 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:22 smithi119 ceph-mon[33714]: osdmap e605: 8 total, 8 up, 8 in 2024-03-24T11:53:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:22 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:23.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:22 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:24.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[30312]: osdmap e606: 8 total, 8 up, 8 in 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[30312]: pgmap v941: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[42466]: osdmap e606: 8 total, 8 up, 8 in 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:24.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:23 smithi114 ceph-mon[42466]: pgmap v941: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:24.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:23 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:24.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:23 smithi119 ceph-mon[33714]: osdmap e606: 8 total, 8 up, 8 in 2024-03-24T11:53:24.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:24.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:23 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:24.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:23 smithi119 ceph-mon[33714]: pgmap v941: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[30312]: osdmap e607: 8 total, 8 up, 8 in 2024-03-24T11:53:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:24.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:53:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[42466]: osdmap e607: 8 total, 8 up, 8 in 2024-03-24T11:53:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:24.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:25.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:25.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:24 smithi119 ceph-mon[33714]: osdmap e607: 8 total, 8 up, 8 in 2024-03-24T11:53:25.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:25.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:26.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]': finished 2024-03-24T11:53:26.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[30312]: osdmap e608: 8 total, 8 up, 8 in 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]: dispatch 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]: dispatch 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[30312]: pgmap v944: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]': finished 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]': finished 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[42466]: osdmap e608: 8 total, 8 up, 8 in 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]: dispatch 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]: dispatch 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[42466]: pgmap v944: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:26.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:25 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]': finished 2024-03-24T11:53:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:25 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-120"}]': finished 2024-03-24T11:53:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:25 smithi119 ceph-mon[33714]: osdmap e608: 8 total, 8 up, 8 in 2024-03-24T11:53:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]: dispatch 2024-03-24T11:53:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:25 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]: dispatch 2024-03-24T11:53:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:25 smithi119 ceph-mon[33714]: pgmap v944: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:25 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:26.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:25 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-120", "mode": "writeback"}]': finished 2024-03-24T11:53:27.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:26 smithi114 ceph-mon[30312]: osdmap e609: 8 total, 8 up, 8 in 2024-03-24T11:53:27.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:27.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:26 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:27.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:26 smithi114 ceph-mon[42466]: osdmap e609: 8 total, 8 up, 8 in 2024-03-24T11:53:27.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:27.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:26 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:26 smithi119 ceph-mon[33714]: osdmap e609: 8 total, 8 up, 8 in 2024-03-24T11:53:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:27.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:26 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:28.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:28.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: osdmap e610: 8 total, 8 up, 8 in 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: pgmap v947: 332 pgs: 332 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]': finished 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[42466]: osdmap e611: 8 total, 8 up, 8 in 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: osdmap e610: 8 total, 8 up, 8 in 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: pgmap v947: 332 pgs: 332 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T11:53:28.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:28.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]': finished 2024-03-24T11:53:28.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:27 smithi114 ceph-mon[30312]: osdmap e611: 8 total, 8 up, 8 in 2024-03-24T11:53:28.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:28.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: osdmap e610: 8 total, 8 up, 8 in 2024-03-24T11:53:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]: dispatch 2024-03-24T11:53:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: pgmap v947: 332 pgs: 332 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T11:53:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-120"}]': finished 2024-03-24T11:53:28.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:27 smithi119 ceph-mon[33714]: osdmap e611: 8 total, 8 up, 8 in 2024-03-24T11:53:30.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:29 smithi114 ceph-mon[42466]: osdmap e612: 8 total, 8 up, 8 in 2024-03-24T11:53:30.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:29 smithi114 ceph-mon[42466]: pgmap v950: 300 pgs: 300 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:53:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:29 smithi114 ceph-mon[30312]: osdmap e612: 8 total, 8 up, 8 in 2024-03-24T11:53:30.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:29 smithi114 ceph-mon[30312]: pgmap v950: 300 pgs: 300 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:53:30.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:29 smithi119 ceph-mon[33714]: osdmap e612: 8 total, 8 up, 8 in 2024-03-24T11:53:30.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:29 smithi119 ceph-mon[33714]: pgmap v950: 300 pgs: 300 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:53:31.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:30 smithi114 ceph-mon[42466]: osdmap e613: 8 total, 8 up, 8 in 2024-03-24T11:53:31.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:31.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:30 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:30 smithi114 ceph-mon[30312]: osdmap e613: 8 total, 8 up, 8 in 2024-03-24T11:53:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:30 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:31.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:30 smithi119 ceph-mon[33714]: osdmap e613: 8 total, 8 up, 8 in 2024-03-24T11:53:31.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:30 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[30312]: osdmap e614: 8 total, 8 up, 8 in 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[30312]: pgmap v953: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[42466]: osdmap e614: 8 total, 8 up, 8 in 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:31 smithi114 ceph-mon[42466]: pgmap v953: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:32.204 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:32.204 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:31 smithi119 ceph-mon[33714]: osdmap e614: 8 total, 8 up, 8 in 2024-03-24T11:53:32.205 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:32.205 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:32.205 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:31 smithi119 ceph-mon[33714]: pgmap v953: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:32.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:53:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:53:33.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:33.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[30312]: osdmap e615: 8 total, 8 up, 8 in 2024-03-24T11:53:33.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[42466]: osdmap e615: 8 total, 8 up, 8 in 2024-03-24T11:53:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:32 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:33.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:32 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:32 smithi119 ceph-mon[33714]: osdmap e615: 8 total, 8 up, 8 in 2024-03-24T11:53:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:32 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:32 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:34.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]': finished 2024-03-24T11:53:34.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: osdmap e616: 8 total, 8 up, 8 in 2024-03-24T11:53:34.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]: dispatch 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]: dispatch 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: pgmap v956: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]': finished 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[30312]: osdmap e617: 8 total, 8 up, 8 in 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]': finished 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: osdmap e616: 8 total, 8 up, 8 in 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]: dispatch 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]: dispatch 2024-03-24T11:53:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: pgmap v956: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]': finished 2024-03-24T11:53:34.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:33 smithi114 ceph-mon[42466]: osdmap e617: 8 total, 8 up, 8 in 2024-03-24T11:53:34.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-122"}]': finished 2024-03-24T11:53:34.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: osdmap e616: 8 total, 8 up, 8 in 2024-03-24T11:53:34.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]: dispatch 2024-03-24T11:53:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]: dispatch 2024-03-24T11:53:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: pgmap v956: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-122", "mode": "writeback"}]': finished 2024-03-24T11:53:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:33 smithi119 ceph-mon[33714]: osdmap e617: 8 total, 8 up, 8 in 2024-03-24T11:53:34.903 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:53:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:53:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:35.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:35.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:34 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:35.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:35.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:34 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:35.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:35.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:34 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:36.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[30312]: osdmap e618: 8 total, 8 up, 8 in 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[30312]: pgmap v959: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[42466]: osdmap e618: 8 total, 8 up, 8 in 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:35 smithi114 ceph-mon[42466]: pgmap v959: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:36.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:35 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:35 smithi119 ceph-mon[33714]: osdmap e618: 8 total, 8 up, 8 in 2024-03-24T11:53:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:35 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:35 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]: dispatch 2024-03-24T11:53:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:35 smithi119 ceph-mon[33714]: pgmap v959: 332 pgs: 332 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:37.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:37 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:37.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:37 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]': finished 2024-03-24T11:53:37.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:37 smithi114 ceph-mon[30312]: osdmap e619: 8 total, 8 up, 8 in 2024-03-24T11:53:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:37 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:37 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]': finished 2024-03-24T11:53:37.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:37 smithi114 ceph-mon[42466]: osdmap e619: 8 total, 8 up, 8 in 2024-03-24T11:53:37.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:37 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:37.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:37 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-122"}]': finished 2024-03-24T11:53:37.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:37 smithi119 ceph-mon[33714]: osdmap e619: 8 total, 8 up, 8 in 2024-03-24T11:53:38.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[30312]: osdmap e620: 8 total, 8 up, 8 in 2024-03-24T11:53:38.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[30312]: pgmap v962: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:38.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[30312]: osdmap e621: 8 total, 8 up, 8 in 2024-03-24T11:53:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:38.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[42466]: osdmap e620: 8 total, 8 up, 8 in 2024-03-24T11:53:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[42466]: pgmap v962: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[42466]: osdmap e621: 8 total, 8 up, 8 in 2024-03-24T11:53:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:38.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:38 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:38.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:38 smithi119 ceph-mon[33714]: osdmap e620: 8 total, 8 up, 8 in 2024-03-24T11:53:38.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:38 smithi119 ceph-mon[33714]: pgmap v962: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:38.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:38 smithi119 ceph-mon[33714]: osdmap e621: 8 total, 8 up, 8 in 2024-03-24T11:53:38.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:38.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:38 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:40.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:39 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:40.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:39 smithi114 ceph-mon[30312]: osdmap e622: 8 total, 8 up, 8 in 2024-03-24T11:53:40.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:39 smithi114 ceph-mon[30312]: pgmap v965: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:39 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:39 smithi114 ceph-mon[42466]: osdmap e622: 8 total, 8 up, 8 in 2024-03-24T11:53:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:39 smithi114 ceph-mon[42466]: pgmap v965: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:40.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:39 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:39 smithi119 ceph-mon[33714]: osdmap e622: 8 total, 8 up, 8 in 2024-03-24T11:53:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:39 smithi119 ceph-mon[33714]: pgmap v965: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:41.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:40 smithi114 ceph-mon[42466]: osdmap e623: 8 total, 8 up, 8 in 2024-03-24T11:53:41.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:41.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:40 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:40 smithi114 ceph-mon[30312]: osdmap e623: 8 total, 8 up, 8 in 2024-03-24T11:53:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:41.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:40 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:41.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:40 smithi119 ceph-mon[33714]: osdmap e623: 8 total, 8 up, 8 in 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:40 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:42.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:42.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[30312]: osdmap e624: 8 total, 8 up, 8 in 2024-03-24T11:53:42.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:42.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:42.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[30312]: pgmap v968: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:42.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[42466]: osdmap e624: 8 total, 8 up, 8 in 2024-03-24T11:53:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[42466]: pgmap v968: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:41 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:42.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:53:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:41 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:41 smithi119 ceph-mon[33714]: osdmap e624: 8 total, 8 up, 8 in 2024-03-24T11:53:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:41 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:41 smithi119 ceph-mon[33714]: pgmap v968: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:42.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:41 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:53:43.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]': finished 2024-03-24T11:53:43.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: osdmap e625: 8 total, 8 up, 8 in 2024-03-24T11:53:43.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]': finished 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: osdmap e625: 8 total, 8 up, 8 in 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:43.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:42 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:43.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-124"}]': finished 2024-03-24T11:53:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: osdmap e625: 8 total, 8 up, 8 in 2024-03-24T11:53:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]: dispatch 2024-03-24T11:53:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]: dispatch 2024-03-24T11:53:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:53:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:53:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:42 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:53:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]': finished 2024-03-24T11:53:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[30312]: osdmap e626: 8 total, 8 up, 8 in 2024-03-24T11:53:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[30312]: pgmap v971: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:53:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]': finished 2024-03-24T11:53:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[42466]: osdmap e626: 8 total, 8 up, 8 in 2024-03-24T11:53:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:43 smithi114 ceph-mon[42466]: pgmap v971: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:53:44.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:43 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:43 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-124", "mode": "writeback"}]': finished 2024-03-24T11:53:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:43 smithi119 ceph-mon[33714]: osdmap e626: 8 total, 8 up, 8 in 2024-03-24T11:53:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:43 smithi119 ceph-mon[33714]: pgmap v971: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:53:44.966 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:53:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:53:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:45.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:44 smithi114 ceph-mon[30312]: osdmap e627: 8 total, 8 up, 8 in 2024-03-24T11:53:45.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:45.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:44 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:45.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:44 smithi114 ceph-mon[42466]: osdmap e627: 8 total, 8 up, 8 in 2024-03-24T11:53:45.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:45.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:44 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:45.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:44 smithi119 ceph-mon[33714]: osdmap e627: 8 total, 8 up, 8 in 2024-03-24T11:53:45.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:45.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:44 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:46.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[30312]: pgmap v973: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:53:46.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:46.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[30312]: osdmap e628: 8 total, 8 up, 8 in 2024-03-24T11:53:46.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:46.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:46.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:46.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]': finished 2024-03-24T11:53:46.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[42466]: pgmap v973: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:53:46.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:46.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[42466]: osdmap e628: 8 total, 8 up, 8 in 2024-03-24T11:53:46.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:46.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:46.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:46.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]': finished 2024-03-24T11:53:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:45 smithi119 ceph-mon[33714]: pgmap v973: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-24T11:53:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:45 smithi119 ceph-mon[33714]: osdmap e628: 8 total, 8 up, 8 in 2024-03-24T11:53:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]: dispatch 2024-03-24T11:53:46.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:45 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:46.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-124"}]': finished 2024-03-24T11:53:47.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:46 smithi114 ceph-mon[30312]: osdmap e629: 8 total, 8 up, 8 in 2024-03-24T11:53:47.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:46 smithi114 ceph-mon[42466]: osdmap e629: 8 total, 8 up, 8 in 2024-03-24T11:53:47.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:46 smithi119 ceph-mon[33714]: osdmap e629: 8 total, 8 up, 8 in 2024-03-24T11:53:48.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[30312]: pgmap v976: 332 pgs: 332 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T11:53:48.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:48.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[30312]: osdmap e630: 8 total, 8 up, 8 in 2024-03-24T11:53:48.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[30312]: osdmap e631: 8 total, 8 up, 8 in 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[42466]: pgmap v976: 332 pgs: 332 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[42466]: osdmap e630: 8 total, 8 up, 8 in 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[42466]: osdmap e631: 8 total, 8 up, 8 in 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:48.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:47 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:48.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:47 smithi119 ceph-mon[33714]: pgmap v976: 332 pgs: 332 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T11:53:48.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:47 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:48.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:47 smithi119 ceph-mon[33714]: osdmap e630: 8 total, 8 up, 8 in 2024-03-24T11:53:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:47 smithi119 ceph-mon[33714]: osdmap e631: 8 total, 8 up, 8 in 2024-03-24T11:53:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:47 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[30312]: osdmap e632: 8 total, 8 up, 8 in 2024-03-24T11:53:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[42466]: osdmap e632: 8 total, 8 up, 8 in 2024-03-24T11:53:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:49.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:53:49.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:49.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:48 smithi119 ceph-mon[33714]: osdmap e632: 8 total, 8 up, 8 in 2024-03-24T11:53:49.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:49.307 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:50.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[30312]: pgmap v980: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:53:50.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[30312]: osdmap e633: 8 total, 8 up, 8 in 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[42466]: pgmap v980: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[42466]: osdmap e633: 8 total, 8 up, 8 in 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:49 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:50.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:49 smithi119 ceph-mon[33714]: pgmap v980: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:53:50.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:49 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:49 smithi119 ceph-mon[33714]: osdmap e633: 8 total, 8 up, 8 in 2024-03-24T11:53:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:49 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:52.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:51 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]': finished 2024-03-24T11:53:52.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:51 smithi119 ceph-mon[33714]: osdmap e634: 8 total, 8 up, 8 in 2024-03-24T11:53:52.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]: dispatch 2024-03-24T11:53:52.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:51 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]: dispatch 2024-03-24T11:53:52.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:51 smithi119 ceph-mon[33714]: pgmap v983: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]': finished 2024-03-24T11:53:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[30312]: osdmap e634: 8 total, 8 up, 8 in 2024-03-24T11:53:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]: dispatch 2024-03-24T11:53:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]: dispatch 2024-03-24T11:53:52.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[30312]: pgmap v983: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-126"}]': finished 2024-03-24T11:53:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[42466]: osdmap e634: 8 total, 8 up, 8 in 2024-03-24T11:53:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]: dispatch 2024-03-24T11:53:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]: dispatch 2024-03-24T11:53:52.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:51 smithi114 ceph-mon[42466]: pgmap v983: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:53:52.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:53:53.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]': finished 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[30312]: osdmap e635: 8 total, 8 up, 8 in 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]': finished 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[42466]: osdmap e635: 8 total, 8 up, 8 in 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:53.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:53.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:52 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:53.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:53:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-126", "mode": "writeback"}]': finished 2024-03-24T11:53:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-mon[33714]: osdmap e635: 8 total, 8 up, 8 in 2024-03-24T11:53:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:53:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:53:53.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:52 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:53:54.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: osdmap e636: 8 total, 8 up, 8 in 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: pgmap v986: 332 pgs: 332 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]': finished 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[30312]: osdmap e637: 8 total, 8 up, 8 in 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:54.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: osdmap e636: 8 total, 8 up, 8 in 2024-03-24T11:53:54.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:54.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:54.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: pgmap v986: 332 pgs: 332 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:54.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:54.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]': finished 2024-03-24T11:53:54.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:53 smithi114 ceph-mon[42466]: osdmap e637: 8 total, 8 up, 8 in 2024-03-24T11:53:54.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:53:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: osdmap e636: 8 total, 8 up, 8 in 2024-03-24T11:53:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]: dispatch 2024-03-24T11:53:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: pgmap v986: 332 pgs: 332 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:53:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-126"}]': finished 2024-03-24T11:53:54.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:53 smithi119 ceph-mon[33714]: osdmap e637: 8 total, 8 up, 8 in 2024-03-24T11:53:54.862 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:53:54.862 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9052 ms) 2024-03-24T11:53:54.862 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-24T11:53:54.862 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:53:54.862 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6047 ms) 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5050 ms) 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25375 ms) 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14058 ms) 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12068 ms) 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-24T11:53:54.863 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: 00000012 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12065 ms) 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: require_osd_release = squid 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6048 ms) 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (554734 ms total) 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1489 ms) 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10600 ms) 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18094 ms) 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-24T11:53:54.864 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30185 ms total) 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7021 ms) 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8063 ms) 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: waiting for scrub... 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: done waiting 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24500 ms) 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9762 ms) 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8050 ms) 2024-03-24T11:53:54.865 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-24T11:53:54.866 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8050 ms) 2024-03-24T11:53:54.866 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-24T11:53:54.866 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10066 ms) 2024-03-24T11:53:54.866 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-24T11:53:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:53:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:53:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:53:56.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:55 smithi114 ceph-mon[42466]: osdmap e638: 8 total, 8 up, 8 in 2024-03-24T11:53:56.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:55 smithi114 ceph-mon[42466]: pgmap v989: 300 pgs: 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:56.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:55 smithi114 ceph-mon[30312]: osdmap e638: 8 total, 8 up, 8 in 2024-03-24T11:53:56.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:55 smithi114 ceph-mon[30312]: pgmap v989: 300 pgs: 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:56.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:55 smithi119 ceph-mon[33714]: osdmap e638: 8 total, 8 up, 8 in 2024-03-24T11:53:56.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:55 smithi119 ceph-mon[33714]: pgmap v989: 300 pgs: 300 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:53:57.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:56 smithi114 ceph-mon[30312]: osdmap e639: 8 total, 8 up, 8 in 2024-03-24T11:53:57.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:57.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:56 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:56 smithi114 ceph-mon[42466]: osdmap e639: 8 total, 8 up, 8 in 2024-03-24T11:53:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:57.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:56 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:57.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:56 smithi119 ceph-mon[33714]: osdmap e639: 8 total, 8 up, 8 in 2024-03-24T11:53:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:57.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:56 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:53:58.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: osdmap e640: 8 total, 8 up, 8 in 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: pgmap v992: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: osdmap e641: 8 total, 8 up, 8 in 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: osdmap e640: 8 total, 8 up, 8 in 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:58.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: pgmap v992: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: osdmap e641: 8 total, 8 up, 8 in 2024-03-24T11:53:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:53:58.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:57 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: osdmap e640: 8 total, 8 up, 8 in 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: pgmap v992: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: osdmap e641: 8 total, 8 up, 8 in 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:53:58.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:57 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:54:00.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]': finished 2024-03-24T11:54:00.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]: dispatch 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[30312]: osdmap e642: 8 total, 8 up, 8 in 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]: dispatch 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[30312]: pgmap v995: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]': finished 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]: dispatch 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[42466]: osdmap e642: 8 total, 8 up, 8 in 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]: dispatch 2024-03-24T11:54:00.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:53:59 smithi114 ceph-mon[42466]: pgmap v995: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:00.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:59 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-128"}]': finished 2024-03-24T11:54:00.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]: dispatch 2024-03-24T11:54:00.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:59 smithi119 ceph-mon[33714]: osdmap e642: 8 total, 8 up, 8 in 2024-03-24T11:54:00.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:59 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]: dispatch 2024-03-24T11:54:00.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:53:59 smithi119 ceph-mon[33714]: pgmap v995: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:01.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:00 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:01.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:00 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]': finished 2024-03-24T11:54:01.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:00 smithi114 ceph-mon[30312]: osdmap e643: 8 total, 8 up, 8 in 2024-03-24T11:54:01.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:00 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:01.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:00 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]': finished 2024-03-24T11:54:01.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:00 smithi114 ceph-mon[42466]: osdmap e643: 8 total, 8 up, 8 in 2024-03-24T11:54:01.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:00 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:01.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:00 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-128", "mode": "writeback"}]': finished 2024-03-24T11:54:01.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:00 smithi119 ceph-mon[33714]: osdmap e643: 8 total, 8 up, 8 in 2024-03-24T11:54:02.139 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:01 smithi114 ceph-mon[30312]: pgmap v997: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:02.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:01 smithi114 ceph-mon[42466]: pgmap v997: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:02.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:01 smithi119 ceph-mon[33714]: pgmap v997: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:02.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:54:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:54:03.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:02 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:03.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:02 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:03.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:02 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:04.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:03 smithi114 ceph-mon[30312]: pgmap v998: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-24T11:54:04.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:04.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:03 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:04.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:03 smithi114 ceph-mon[42466]: pgmap v998: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-24T11:54:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:04.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:03 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:04.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:03 smithi119 ceph-mon[33714]: pgmap v998: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-24T11:54:04.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:04.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:03 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:54:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:54:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:05.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:05.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:05 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:05.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:05.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:05 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:05.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:05.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:05 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:06.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:06 smithi119 ceph-mon[33714]: pgmap v999: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 288 B/s wr, 1 op/s 2024-03-24T11:54:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:06 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:06 smithi119 ceph-mon[33714]: osdmap e644: 8 total, 8 up, 8 in 2024-03-24T11:54:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:54:06.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:06 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:54:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[30312]: pgmap v999: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 288 B/s wr, 1 op/s 2024-03-24T11:54:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[30312]: osdmap e644: 8 total, 8 up, 8 in 2024-03-24T11:54:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:54:06.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:54:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[42466]: pgmap v999: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 288 B/s wr, 1 op/s 2024-03-24T11:54:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[42466]: osdmap e644: 8 total, 8 up, 8 in 2024-03-24T11:54:06.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:54:06.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:06 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]: dispatch 2024-03-24T11:54:07.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:07 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:07.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:07 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]': finished 2024-03-24T11:54:07.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:07 smithi119 ceph-mon[33714]: osdmap e645: 8 total, 8 up, 8 in 2024-03-24T11:54:07.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:07 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:07.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:07 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]': finished 2024-03-24T11:54:07.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:07 smithi114 ceph-mon[30312]: osdmap e645: 8 total, 8 up, 8 in 2024-03-24T11:54:07.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:07 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:07.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:07 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-128"}]': finished 2024-03-24T11:54:07.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:07 smithi114 ceph-mon[42466]: osdmap e645: 8 total, 8 up, 8 in 2024-03-24T11:54:08.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:08 smithi119 ceph-mon[33714]: pgmap v1002: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 288 B/s wr, 2 op/s 2024-03-24T11:54:08.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:08 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:08.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:08 smithi119 ceph-mon[33714]: osdmap e646: 8 total, 8 up, 8 in 2024-03-24T11:54:08.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:08 smithi114 ceph-mon[30312]: pgmap v1002: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 288 B/s wr, 2 op/s 2024-03-24T11:54:08.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:08 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:08.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:08 smithi114 ceph-mon[30312]: osdmap e646: 8 total, 8 up, 8 in 2024-03-24T11:54:08.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:08 smithi114 ceph-mon[42466]: pgmap v1002: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 288 B/s wr, 2 op/s 2024-03-24T11:54:08.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:08 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:08.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:08 smithi114 ceph-mon[42466]: osdmap e646: 8 total, 8 up, 8 in 2024-03-24T11:54:09.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:09 smithi119 ceph-mon[33714]: osdmap e647: 8 total, 8 up, 8 in 2024-03-24T11:54:09.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:09.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:09 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:09.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:09 smithi114 ceph-mon[30312]: osdmap e647: 8 total, 8 up, 8 in 2024-03-24T11:54:09.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:09.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:09 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:09.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:09 smithi114 ceph-mon[42466]: osdmap e647: 8 total, 8 up, 8 in 2024-03-24T11:54:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:09.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:09 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[42466]: pgmap v1005: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[42466]: osdmap e648: 8 total, 8 up, 8 in 2024-03-24T11:54:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:10.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[30312]: pgmap v1005: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[30312]: osdmap e648: 8 total, 8 up, 8 in 2024-03-24T11:54:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:10.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:10 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:10 smithi119 ceph-mon[33714]: pgmap v1005: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:54:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:10 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:10 smithi119 ceph-mon[33714]: osdmap e648: 8 total, 8 up, 8 in 2024-03-24T11:54:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:10.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:10 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:11.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:11.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: osdmap e649: 8 total, 8 up, 8 in 2024-03-24T11:54:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]': finished 2024-03-24T11:54:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: osdmap e650: 8 total, 8 up, 8 in 2024-03-24T11:54:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]: dispatch 2024-03-24T11:54:11.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]: dispatch 2024-03-24T11:54:11.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:11.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: osdmap e649: 8 total, 8 up, 8 in 2024-03-24T11:54:11.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:11.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:11.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]': finished 2024-03-24T11:54:11.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: osdmap e650: 8 total, 8 up, 8 in 2024-03-24T11:54:11.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]: dispatch 2024-03-24T11:54:11.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:11 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]: dispatch 2024-03-24T11:54:11.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: osdmap e649: 8 total, 8 up, 8 in 2024-03-24T11:54:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-130"}]': finished 2024-03-24T11:54:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: osdmap e650: 8 total, 8 up, 8 in 2024-03-24T11:54:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]: dispatch 2024-03-24T11:54:11.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:11 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]: dispatch 2024-03-24T11:54:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[30312]: pgmap v1008: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]': finished 2024-03-24T11:54:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[30312]: osdmap e651: 8 total, 8 up, 8 in 2024-03-24T11:54:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[42466]: pgmap v1008: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]': finished 2024-03-24T11:54:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:12 smithi114 ceph-mon[42466]: osdmap e651: 8 total, 8 up, 8 in 2024-03-24T11:54:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:12 smithi119 ceph-mon[33714]: pgmap v1008: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:12 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:12 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-130", "mode": "writeback"}]': finished 2024-03-24T11:54:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:12 smithi119 ceph-mon[33714]: osdmap e651: 8 total, 8 up, 8 in 2024-03-24T11:54:12.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:54:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:54:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:13 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:13 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:13 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:13 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[42466]: pgmap v1011: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[42466]: osdmap e652: 8 total, 8 up, 8 in 2024-03-24T11:54:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[30312]: pgmap v1011: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[30312]: osdmap e652: 8 total, 8 up, 8 in 2024-03-24T11:54:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:14 smithi119 ceph-mon[33714]: pgmap v1011: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:14 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:14 smithi119 ceph-mon[33714]: osdmap e652: 8 total, 8 up, 8 in 2024-03-24T11:54:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:14.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:14 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]: dispatch 2024-03-24T11:54:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:54:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:54:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:15.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:15 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:15.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:15 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]': finished 2024-03-24T11:54:15.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:15 smithi114 ceph-mon[30312]: osdmap e653: 8 total, 8 up, 8 in 2024-03-24T11:54:15.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:15 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:15 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]': finished 2024-03-24T11:54:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:15 smithi114 ceph-mon[42466]: osdmap e653: 8 total, 8 up, 8 in 2024-03-24T11:54:15.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:15 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:15.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:15 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-130"}]': finished 2024-03-24T11:54:15.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:15 smithi119 ceph-mon[33714]: osdmap e653: 8 total, 8 up, 8 in 2024-03-24T11:54:16.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:16 smithi114 ceph-mon[30312]: pgmap v1014: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:16.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:16 smithi114 ceph-mon[30312]: osdmap e654: 8 total, 8 up, 8 in 2024-03-24T11:54:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:16 smithi114 ceph-mon[42466]: pgmap v1014: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:16 smithi114 ceph-mon[42466]: osdmap e654: 8 total, 8 up, 8 in 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:16 smithi119 ceph-mon[33714]: pgmap v1014: 332 pgs: 332 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:16 smithi119 ceph-mon[33714]: osdmap e654: 8 total, 8 up, 8 in 2024-03-24T11:54:17.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[42466]: osdmap e655: 8 total, 8 up, 8 in 2024-03-24T11:54:17.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:17.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:17.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[42466]: osdmap e656: 8 total, 8 up, 8 in 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[30312]: osdmap e655: 8 total, 8 up, 8 in 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[30312]: osdmap e656: 8 total, 8 up, 8 in 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:17 smithi119 ceph-mon[33714]: osdmap e655: 8 total, 8 up, 8 in 2024-03-24T11:54:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:17 smithi119 ceph-mon[33714]: osdmap e656: 8 total, 8 up, 8 in 2024-03-24T11:54:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[42466]: pgmap v1017: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[42466]: osdmap e657: 8 total, 8 up, 8 in 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[30312]: pgmap v1017: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[30312]: osdmap e657: 8 total, 8 up, 8 in 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:18 smithi119 ceph-mon[33714]: pgmap v1017: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:18 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:18 smithi119 ceph-mon[33714]: osdmap e657: 8 total, 8 up, 8 in 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[42466]: pgmap v1020: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[42466]: osdmap e658: 8 total, 8 up, 8 in 2024-03-24T11:54:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]: dispatch 2024-03-24T11:54:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]: dispatch 2024-03-24T11:54:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[30312]: pgmap v1020: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[30312]: osdmap e658: 8 total, 8 up, 8 in 2024-03-24T11:54:20.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]: dispatch 2024-03-24T11:54:20.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:19 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]: dispatch 2024-03-24T11:54:20.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:20.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:19 smithi119 ceph-mon[33714]: pgmap v1020: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:54:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:19 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:19 smithi119 ceph-mon[33714]: osdmap e658: 8 total, 8 up, 8 in 2024-03-24T11:54:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]: dispatch 2024-03-24T11:54:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:19 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]: dispatch 2024-03-24T11:54:21.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:20 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:21.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:20 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]': finished 2024-03-24T11:54:21.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:20 smithi114 ceph-mon[30312]: osdmap e659: 8 total, 8 up, 8 in 2024-03-24T11:54:21.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:20 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:21.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:20 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]': finished 2024-03-24T11:54:21.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:20 smithi114 ceph-mon[42466]: osdmap e659: 8 total, 8 up, 8 in 2024-03-24T11:54:21.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:20 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:20 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-132", "mode": "writeback"}]': finished 2024-03-24T11:54:21.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:20 smithi119 ceph-mon[33714]: osdmap e659: 8 total, 8 up, 8 in 2024-03-24T11:54:22.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:22 smithi114 ceph-mon[42466]: pgmap v1023: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:22.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:22 smithi114 ceph-mon[42466]: osdmap e660: 8 total, 8 up, 8 in 2024-03-24T11:54:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:22 smithi114 ceph-mon[30312]: pgmap v1023: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:22.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:22 smithi114 ceph-mon[30312]: osdmap e660: 8 total, 8 up, 8 in 2024-03-24T11:54:22.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:22 smithi119 ceph-mon[33714]: pgmap v1023: 332 pgs: 32 creating+peering, 300 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:22.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:22 smithi119 ceph-mon[33714]: osdmap e660: 8 total, 8 up, 8 in 2024-03-24T11:54:22.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:54:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:54:23.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[30312]: osdmap e661: 8 total, 8 up, 8 in 2024-03-24T11:54:23.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:23.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:23.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:23.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[42466]: osdmap e661: 8 total, 8 up, 8 in 2024-03-24T11:54:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:23 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:23 smithi119 ceph-mon[33714]: osdmap e661: 8 total, 8 up, 8 in 2024-03-24T11:54:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:23 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:23 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[42466]: pgmap v1026: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:54:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[42466]: osdmap e662: 8 total, 8 up, 8 in 2024-03-24T11:54:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[30312]: pgmap v1026: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:54:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[30312]: osdmap e662: 8 total, 8 up, 8 in 2024-03-24T11:54:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:24 smithi119 ceph-mon[33714]: pgmap v1026: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:54:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:24 smithi119 ceph-mon[33714]: osdmap e662: 8 total, 8 up, 8 in 2024-03-24T11:54:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:54:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:54:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:25.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:25 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:25 smithi114 ceph-mon[30312]: osdmap e663: 8 total, 8 up, 8 in 2024-03-24T11:54:25.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:25 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:25.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:25 smithi114 ceph-mon[42466]: osdmap e663: 8 total, 8 up, 8 in 2024-03-24T11:54:25.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:25 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:25.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:25 smithi119 ceph-mon[33714]: osdmap e663: 8 total, 8 up, 8 in 2024-03-24T11:54:26.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[30312]: pgmap v1029: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:54:26.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[30312]: osdmap e664: 8 total, 8 up, 8 in 2024-03-24T11:54:26.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:26.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[42466]: pgmap v1029: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:54:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[42466]: osdmap e664: 8 total, 8 up, 8 in 2024-03-24T11:54:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:26 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:26 smithi119 ceph-mon[33714]: pgmap v1029: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:54:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:26 smithi119 ceph-mon[33714]: osdmap e664: 8 total, 8 up, 8 in 2024-03-24T11:54:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:26 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:27.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:27.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[30312]: osdmap e665: 8 total, 8 up, 8 in 2024-03-24T11:54:27.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:27.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:27.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:27.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[42466]: osdmap e665: 8 total, 8 up, 8 in 2024-03-24T11:54:27.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:27.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:27.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:27 smithi119 ceph-mon[33714]: osdmap e665: 8 total, 8 up, 8 in 2024-03-24T11:54:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]: dispatch 2024-03-24T11:54:28.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[30312]: pgmap v1032: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-24T11:54:28.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:28.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[30312]: osdmap e666: 8 total, 8 up, 8 in 2024-03-24T11:54:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[30312]: osdmap e667: 8 total, 8 up, 8 in 2024-03-24T11:54:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[42466]: pgmap v1032: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-24T11:54:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[42466]: osdmap e666: 8 total, 8 up, 8 in 2024-03-24T11:54:28.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:28 smithi114 ceph-mon[42466]: osdmap e667: 8 total, 8 up, 8 in 2024-03-24T11:54:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:28 smithi119 ceph-mon[33714]: pgmap v1032: 332 pgs: 332 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-24T11:54:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:28 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:28 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-132"}]': finished 2024-03-24T11:54:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:28 smithi119 ceph-mon[33714]: osdmap e666: 8 total, 8 up, 8 in 2024-03-24T11:54:28.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:28 smithi119 ceph-mon[33714]: osdmap e667: 8 total, 8 up, 8 in 2024-03-24T11:54:30.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[30312]: osdmap e668: 8 total, 8 up, 8 in 2024-03-24T11:54:30.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:30.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:30.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[30312]: pgmap v1036: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 266 B/s wr, 2 op/s 2024-03-24T11:54:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[42466]: osdmap e668: 8 total, 8 up, 8 in 2024-03-24T11:54:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:30.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:29 smithi114 ceph-mon[42466]: pgmap v1036: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 266 B/s wr, 2 op/s 2024-03-24T11:54:30.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:29 smithi119 ceph-mon[33714]: osdmap e668: 8 total, 8 up, 8 in 2024-03-24T11:54:30.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:30.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:29 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:30.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:29 smithi119 ceph-mon[33714]: pgmap v1036: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 266 B/s wr, 2 op/s 2024-03-24T11:54:31.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:31.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[30312]: osdmap e669: 8 total, 8 up, 8 in 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[42466]: osdmap e669: 8 total, 8 up, 8 in 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:31.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:30 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:31.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:30 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:31.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:30 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:30 smithi119 ceph-mon[33714]: osdmap e669: 8 total, 8 up, 8 in 2024-03-24T11:54:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:31.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:30 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[30312]: osdmap e670: 8 total, 8 up, 8 in 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[30312]: pgmap v1039: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]': finished 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[42466]: osdmap e670: 8 total, 8 up, 8 in 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:32.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:32.141 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[42466]: pgmap v1039: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:32.141 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]': finished 2024-03-24T11:54:32.166 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:32.166 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:31 smithi119 ceph-mon[33714]: osdmap e670: 8 total, 8 up, 8 in 2024-03-24T11:54:32.166 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:32.166 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:32.166 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:31 smithi119 ceph-mon[33714]: pgmap v1039: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:32.166 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-134"}]': finished 2024-03-24T11:54:32.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:54:33.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[30312]: osdmap e671: 8 total, 8 up, 8 in 2024-03-24T11:54:33.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]: dispatch 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]: dispatch 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]': finished 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[30312]: osdmap e672: 8 total, 8 up, 8 in 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[42466]: osdmap e671: 8 total, 8 up, 8 in 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]: dispatch 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]: dispatch 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:33.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]': finished 2024-03-24T11:54:33.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:32 smithi114 ceph-mon[42466]: osdmap e672: 8 total, 8 up, 8 in 2024-03-24T11:54:33.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-mon[33714]: osdmap e671: 8 total, 8 up, 8 in 2024-03-24T11:54:33.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]: dispatch 2024-03-24T11:54:33.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]: dispatch 2024-03-24T11:54:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-134", "mode": "writeback"}]': finished 2024-03-24T11:54:33.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:32 smithi119 ceph-mon[33714]: osdmap e672: 8 total, 8 up, 8 in 2024-03-24T11:54:34.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:34.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:33 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:34.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:33 smithi114 ceph-mon[30312]: pgmap v1042: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:33 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:34.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:33 smithi114 ceph-mon[42466]: pgmap v1042: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:33 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:34.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:33 smithi119 ceph-mon[33714]: pgmap v1042: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:54:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[42466]: osdmap e673: 8 total, 8 up, 8 in 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[30312]: osdmap e673: 8 total, 8 up, 8 in 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:34.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:34.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:35.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:34 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:35.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:34 smithi119 ceph-mon[33714]: osdmap e673: 8 total, 8 up, 8 in 2024-03-24T11:54:35.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:34 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]: dispatch 2024-03-24T11:54:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:35.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:36.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:36.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]': finished 2024-03-24T11:54:36.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[42466]: osdmap e674: 8 total, 8 up, 8 in 2024-03-24T11:54:36.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[42466]: pgmap v1045: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]': finished 2024-03-24T11:54:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[30312]: osdmap e674: 8 total, 8 up, 8 in 2024-03-24T11:54:36.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:35 smithi114 ceph-mon[30312]: pgmap v1045: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:36.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:35 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:36.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:35 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-134"}]': finished 2024-03-24T11:54:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:35 smithi119 ceph-mon[33714]: osdmap e674: 8 total, 8 up, 8 in 2024-03-24T11:54:36.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:35 smithi119 ceph-mon[33714]: pgmap v1045: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:37.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:36 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:37.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:36 smithi114 ceph-mon[30312]: osdmap e675: 8 total, 8 up, 8 in 2024-03-24T11:54:37.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:36 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:37.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:36 smithi114 ceph-mon[42466]: osdmap e675: 8 total, 8 up, 8 in 2024-03-24T11:54:37.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:36 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:37.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:36 smithi119 ceph-mon[33714]: osdmap e675: 8 total, 8 up, 8 in 2024-03-24T11:54:38.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[30312]: osdmap e676: 8 total, 8 up, 8 in 2024-03-24T11:54:38.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[30312]: pgmap v1048: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[30312]: osdmap e677: 8 total, 8 up, 8 in 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[42466]: osdmap e676: 8 total, 8 up, 8 in 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[42466]: pgmap v1048: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:54:38.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[42466]: osdmap e677: 8 total, 8 up, 8 in 2024-03-24T11:54:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:37 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:38.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:37 smithi119 ceph-mon[33714]: osdmap e676: 8 total, 8 up, 8 in 2024-03-24T11:54:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:37 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:37 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:37 smithi119 ceph-mon[33714]: pgmap v1048: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:54:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:37 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:37 smithi119 ceph-mon[33714]: osdmap e677: 8 total, 8 up, 8 in 2024-03-24T11:54:38.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:37 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:39.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[30312]: osdmap e678: 8 total, 8 up, 8 in 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[42466]: osdmap e678: 8 total, 8 up, 8 in 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:38 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:39.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:39.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:38 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:39.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:38 smithi119 ceph-mon[33714]: osdmap e678: 8 total, 8 up, 8 in 2024-03-24T11:54:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:38 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:39.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:38 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:40.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[30312]: pgmap v1051: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:54:40.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]': finished 2024-03-24T11:54:40.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[30312]: osdmap e679: 8 total, 8 up, 8 in 2024-03-24T11:54:40.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]: dispatch 2024-03-24T11:54:40.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]: dispatch 2024-03-24T11:54:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[42466]: pgmap v1051: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:54:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]': finished 2024-03-24T11:54:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[42466]: osdmap e679: 8 total, 8 up, 8 in 2024-03-24T11:54:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]: dispatch 2024-03-24T11:54:40.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:39 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]: dispatch 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:39 smithi119 ceph-mon[33714]: pgmap v1051: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:39 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-136"}]': finished 2024-03-24T11:54:40.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:39 smithi119 ceph-mon[33714]: osdmap e679: 8 total, 8 up, 8 in 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:39 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]: dispatch 2024-03-24T11:54:40.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:39 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]: dispatch 2024-03-24T11:54:41.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:40 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:41.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:40 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]': finished 2024-03-24T11:54:41.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:40 smithi114 ceph-mon[30312]: osdmap e680: 8 total, 8 up, 8 in 2024-03-24T11:54:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:40 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:40 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]': finished 2024-03-24T11:54:41.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:40 smithi114 ceph-mon[42466]: osdmap e680: 8 total, 8 up, 8 in 2024-03-24T11:54:41.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:40 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:54:41.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:40 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-136", "mode": "writeback"}]': finished 2024-03-24T11:54:41.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:40 smithi119 ceph-mon[33714]: osdmap e680: 8 total, 8 up, 8 in 2024-03-24T11:54:42.178 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:42.178 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:41 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:42.179 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:41 smithi119 ceph-mon[33714]: pgmap v1054: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:42.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:42.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:41 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:42.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:41 smithi114 ceph-mon[30312]: pgmap v1054: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:41 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:42.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:41 smithi114 ceph-mon[42466]: pgmap v1054: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:42.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:54:43.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:43.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[30312]: osdmap e681: 8 total, 8 up, 8 in 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[42466]: osdmap e681: 8 total, 8 up, 8 in 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:43.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:43.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:42 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:43.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:54:43.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-mon[33714]: osdmap e681: 8 total, 8 up, 8 in 2024-03-24T11:54:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]: dispatch 2024-03-24T11:54:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:54:43.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:42 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]': finished 2024-03-24T11:54:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[42466]: osdmap e682: 8 total, 8 up, 8 in 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[42466]: pgmap v1057: 332 pgs: 332 active+clean; 4.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]': finished 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[30312]: osdmap e682: 8 total, 8 up, 8 in 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[30312]: pgmap v1057: 332 pgs: 332 active+clean; 4.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:44.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:43 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:44.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:43 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:54:44.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:43 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-136"}]': finished 2024-03-24T11:54:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:43 smithi119 ceph-mon[33714]: osdmap e682: 8 total, 8 up, 8 in 2024-03-24T11:54:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:43 smithi119 ceph-mon[33714]: pgmap v1057: 332 pgs: 332 active+clean; 4.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-24T11:54:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:43 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:54:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:43 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:54:44.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:43 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:44.933 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:54:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:54:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:45.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:44 smithi114 ceph-mon[30312]: osdmap e683: 8 total, 8 up, 8 in 2024-03-24T11:54:45.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:44 smithi114 ceph-mon[42466]: osdmap e683: 8 total, 8 up, 8 in 2024-03-24T11:54:45.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:44 smithi119 ceph-mon[33714]: osdmap e683: 8 total, 8 up, 8 in 2024-03-24T11:54:46.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[30312]: osdmap e684: 8 total, 8 up, 8 in 2024-03-24T11:54:46.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:46.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:46.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[30312]: pgmap v1060: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-24T11:54:46.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[42466]: osdmap e684: 8 total, 8 up, 8 in 2024-03-24T11:54:46.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:46.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:46.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:45 smithi114 ceph-mon[42466]: pgmap v1060: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-24T11:54:46.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:45 smithi119 ceph-mon[33714]: osdmap e684: 8 total, 8 up, 8 in 2024-03-24T11:54:46.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:46.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:45 smithi119 ceph-mon[33714]: pgmap v1060: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[30312]: osdmap e685: 8 total, 8 up, 8 in 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[42466]: osdmap e685: 8 total, 8 up, 8 in 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:47.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:46 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:47.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:46 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:47.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:46 smithi119 ceph-mon[33714]: osdmap e685: 8 total, 8 up, 8 in 2024-03-24T11:54:47.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:47.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:46 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: osdmap e686: 8 total, 8 up, 8 in 2024-03-24T11:54:48.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:54:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:54:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: pgmap v1063: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:54:48.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[30312]: osdmap e687: 8 total, 8 up, 8 in 2024-03-24T11:54:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: osdmap e686: 8 total, 8 up, 8 in 2024-03-24T11:54:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:54:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:54:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: pgmap v1063: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:48.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:48.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:54:48.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:47 smithi114 ceph-mon[42466]: osdmap e687: 8 total, 8 up, 8 in 2024-03-24T11:54:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: osdmap e686: 8 total, 8 up, 8 in 2024-03-24T11:54:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:54:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:54:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: pgmap v1063: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:48.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:54:48.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:47 smithi119 ceph-mon[33714]: osdmap e687: 8 total, 8 up, 8 in 2024-03-24T11:54:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:54:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:54:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: osdmap e688: 8 total, 8 up, 8 in 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: osdmap e688: 8 total, 8 up, 8 in 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:54:49.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:54:49.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:54:49.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:54:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:54:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:54:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:54:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: osdmap e688: 8 total, 8 up, 8 in 2024-03-24T11:54:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:54:49.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-24T11:54:50.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:49 smithi114 ceph-mon[30312]: pgmap v1066: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:50.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:49 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:54:50.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:49 smithi114 ceph-mon[30312]: osdmap e689: 8 total, 8 up, 8 in 2024-03-24T11:54:50.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:49 smithi114 ceph-mon[42466]: pgmap v1066: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:50.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:49 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:54:50.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:49 smithi114 ceph-mon[42466]: osdmap e689: 8 total, 8 up, 8 in 2024-03-24T11:54:50.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:49 smithi119 ceph-mon[33714]: pgmap v1066: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:49 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-24T11:54:50.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:49 smithi119 ceph-mon[33714]: osdmap e689: 8 total, 8 up, 8 in 2024-03-24T11:54:51.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:51.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:51 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:51.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]: dispatch 2024-03-24T11:54:51.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:51 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]: dispatch 2024-03-24T11:54:51.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:51.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]: dispatch 2024-03-24T11:54:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]: dispatch 2024-03-24T11:54:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:54:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]: dispatch 2024-03-24T11:54:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:51 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]: dispatch 2024-03-24T11:54:52.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:52 smithi119 ceph-mon[33714]: pgmap v1068: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:52.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:52 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]': finished 2024-03-24T11:54:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:52 smithi119 ceph-mon[33714]: osdmap e690: 8 total, 8 up, 8 in 2024-03-24T11:54:52.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:54:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:54:52.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:52 smithi114 ceph-mon[30312]: pgmap v1068: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:52.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:52 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]': finished 2024-03-24T11:54:52.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:52 smithi114 ceph-mon[30312]: osdmap e690: 8 total, 8 up, 8 in 2024-03-24T11:54:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:52 smithi114 ceph-mon[42466]: pgmap v1068: 332 pgs: 332 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:54:52.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:52 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-138"}]': finished 2024-03-24T11:54:52.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:52 smithi114 ceph-mon[42466]: osdmap e690: 8 total, 8 up, 8 in 2024-03-24T11:54:53.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:53 smithi119 ceph-mon[33714]: osdmap e691: 8 total, 8 up, 8 in 2024-03-24T11:54:53.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:53 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:53 smithi119 ceph-mon[33714]: osdmap e692: 8 total, 8 up, 8 in 2024-03-24T11:54:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:53.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:53.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[30312]: osdmap e691: 8 total, 8 up, 8 in 2024-03-24T11:54:53.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:53.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[30312]: osdmap e692: 8 total, 8 up, 8 in 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[42466]: osdmap e691: 8 total, 8 up, 8 in 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[42466]: osdmap e692: 8 total, 8 up, 8 in 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:54:54.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[42466]: pgmap v1071: 300 pgs: 300 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:54.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:54.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[42466]: osdmap e693: 8 total, 8 up, 8 in 2024-03-24T11:54:54.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:54.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:54.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[30312]: pgmap v1071: 300 pgs: 300 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:54.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[30312]: osdmap e693: 8 total, 8 up, 8 in 2024-03-24T11:54:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:54.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:54 smithi119 ceph-mon[33714]: pgmap v1071: 300 pgs: 300 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:54 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:54:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:54 smithi119 ceph-mon[33714]: osdmap e693: 8 total, 8 up, 8 in 2024-03-24T11:54:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:54 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:54:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:54:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:54:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:54:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[42466]: pgmap v1074: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[42466]: osdmap e694: 8 total, 8 up, 8 in 2024-03-24T11:54:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:54:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:54:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[30312]: pgmap v1074: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[30312]: osdmap e694: 8 total, 8 up, 8 in 2024-03-24T11:54:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:54:56.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:56 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:54:56.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:56 smithi119 ceph-mon[33714]: pgmap v1074: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:56.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:56 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:54:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:56 smithi119 ceph-mon[33714]: osdmap e694: 8 total, 8 up, 8 in 2024-03-24T11:54:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:56 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:54:56.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:56 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-24T11:54:57.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]': finished 2024-03-24T11:54:57.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[30312]: osdmap e695: 8 total, 8 up, 8 in 2024-03-24T11:54:57.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:54:57.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:54:57.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]': finished 2024-03-24T11:54:57.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]': finished 2024-03-24T11:54:57.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[42466]: osdmap e695: 8 total, 8 up, 8 in 2024-03-24T11:54:57.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:54:57.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:54:57.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:57 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]': finished 2024-03-24T11:54:57.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:57 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_count","val": "3"}]': finished 2024-03-24T11:54:57.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:57 smithi119 ceph-mon[33714]: osdmap e695: 8 total, 8 up, 8 in 2024-03-24T11:54:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:57 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:54:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:57 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-24T11:54:57.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:57 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_period","val": "3"}]': finished 2024-03-24T11:54:58.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[30312]: pgmap v1077: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:58.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[30312]: osdmap e696: 8 total, 8 up, 8 in 2024-03-24T11:54:58.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:54:58.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:54:58.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:54:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[42466]: pgmap v1077: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[42466]: osdmap e696: 8 total, 8 up, 8 in 2024-03-24T11:54:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:54:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:54:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:58 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:54:58.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:58 smithi119 ceph-mon[33714]: pgmap v1077: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:54:58.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:58 smithi119 ceph-mon[33714]: osdmap e696: 8 total, 8 up, 8 in 2024-03-24T11:54:58.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:58 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:54:58.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:58 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:54:58.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:58 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:54:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:59 smithi114 ceph-mon[30312]: osdmap e697: 8 total, 8 up, 8 in 2024-03-24T11:54:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:59 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:54:59.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:54:59 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:54:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:59 smithi114 ceph-mon[42466]: osdmap e697: 8 total, 8 up, 8 in 2024-03-24T11:54:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:59 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:54:59.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:54:59 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:54:59.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:59 smithi119 ceph-mon[33714]: osdmap e697: 8 total, 8 up, 8 in 2024-03-24T11:54:59.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:59 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:54:59.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:54:59 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-24T11:55:00.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:00 smithi114 ceph-mon[30312]: pgmap v1080: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:00.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:00 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-24T11:55:00.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:00 smithi114 ceph-mon[30312]: osdmap e698: 8 total, 8 up, 8 in 2024-03-24T11:55:00.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:00 smithi114 ceph-mon[42466]: pgmap v1080: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:00.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:00 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-24T11:55:00.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:00 smithi114 ceph-mon[42466]: osdmap e698: 8 total, 8 up, 8 in 2024-03-24T11:55:00.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:00 smithi119 ceph-mon[33714]: pgmap v1080: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:00.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:00 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-24T11:55:00.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:00 smithi119 ceph-mon[33714]: osdmap e698: 8 total, 8 up, 8 in 2024-03-24T11:55:01.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:01 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "308.10", "id": [5, 6]}]: dispatch 2024-03-24T11:55:01.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:01 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "308.10", "id": [5, 6]}]: dispatch 2024-03-24T11:55:01.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:01 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "308.10", "id": [5, 6]}]: dispatch 2024-03-24T11:55:02.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:02 smithi114 ceph-mon[30312]: pgmap v1082: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:55:02.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "308.10", "id": [5, 6]}]': finished 2024-03-24T11:55:02.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:02 smithi114 ceph-mon[30312]: osdmap e699: 8 total, 8 up, 8 in 2024-03-24T11:55:02.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:02 smithi114 ceph-mon[42466]: pgmap v1082: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:55:02.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "308.10", "id": [5, 6]}]': finished 2024-03-24T11:55:02.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:02 smithi114 ceph-mon[42466]: osdmap e699: 8 total, 8 up, 8 in 2024-03-24T11:55:02.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:02 smithi119 ceph-mon[33714]: pgmap v1082: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:55:02.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "308.10", "id": [5, 6]}]': finished 2024-03-24T11:55:02.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:02 smithi119 ceph-mon[33714]: osdmap e699: 8 total, 8 up, 8 in 2024-03-24T11:55:02.455 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:55:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:55:03.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:03 smithi119 ceph-mon[33714]: osdmap e700: 8 total, 8 up, 8 in 2024-03-24T11:55:03.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:03.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:03 smithi114 ceph-mon[30312]: osdmap e700: 8 total, 8 up, 8 in 2024-03-24T11:55:03.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:03.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:03 smithi114 ceph-mon[42466]: osdmap e700: 8 total, 8 up, 8 in 2024-03-24T11:55:03.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:04 smithi114 ceph-mon[30312]: pgmap v1085: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-24T11:55:04.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:04 smithi114 ceph-mon[42466]: pgmap v1085: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-24T11:55:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:04 smithi119 ceph-mon[33714]: pgmap v1085: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-24T11:55:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:55:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:55:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:06 smithi114 ceph-mon[30312]: pgmap v1086: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T11:55:06.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:06 smithi114 ceph-mon[42466]: pgmap v1086: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T11:55:06.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:06 smithi119 ceph-mon[33714]: pgmap v1086: 332 pgs: 332 active+clean; 4.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T11:55:08.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:08 smithi114 ceph-mon[30312]: pgmap v1087: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-24T11:55:08.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:08 smithi114 ceph-mon[42466]: pgmap v1087: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-24T11:55:08.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:08 smithi119 ceph-mon[33714]: pgmap v1087: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-24T11:55:10.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:10 smithi114 ceph-mon[30312]: pgmap v1088: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-03-24T11:55:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:10 smithi114 ceph-mon[42466]: pgmap v1088: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-03-24T11:55:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:10 smithi119 ceph-mon[33714]: pgmap v1088: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-03-24T11:55:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[30312]: pgmap v1089: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.7 KiB/s wr, 2 op/s 2024-03-24T11:55:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]: dispatch 2024-03-24T11:55:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]: dispatch 2024-03-24T11:55:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[42466]: pgmap v1089: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.7 KiB/s wr, 2 op/s 2024-03-24T11:55:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]: dispatch 2024-03-24T11:55:12.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:12 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]: dispatch 2024-03-24T11:55:12.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:55:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:55:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:12 smithi119 ceph-mon[33714]: pgmap v1089: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.7 KiB/s wr, 2 op/s 2024-03-24T11:55:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:12 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]: dispatch 2024-03-24T11:55:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:12 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]: dispatch 2024-03-24T11:55:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:13 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]': finished 2024-03-24T11:55:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:13 smithi114 ceph-mon[42466]: osdmap e701: 8 total, 8 up, 8 in 2024-03-24T11:55:13.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:13 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]': finished 2024-03-24T11:55:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:13 smithi114 ceph-mon[30312]: osdmap e701: 8 total, 8 up, 8 in 2024-03-24T11:55:13.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:13 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-140"}]': finished 2024-03-24T11:55:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:13 smithi119 ceph-mon[33714]: osdmap e701: 8 total, 8 up, 8 in 2024-03-24T11:55:13.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:14 smithi114 ceph-mon[42466]: pgmap v1091: 332 pgs: 332 active+clean; 4.5 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 2 op/s 2024-03-24T11:55:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:14 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:14 smithi114 ceph-mon[42466]: osdmap e702: 8 total, 8 up, 8 in 2024-03-24T11:55:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:14 smithi114 ceph-mon[30312]: pgmap v1091: 332 pgs: 332 active+clean; 4.5 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 2 op/s 2024-03-24T11:55:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:14 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:14.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:14 smithi114 ceph-mon[30312]: osdmap e702: 8 total, 8 up, 8 in 2024-03-24T11:55:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:14 smithi119 ceph-mon[33714]: pgmap v1091: 332 pgs: 332 active+clean; 4.5 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 2 op/s 2024-03-24T11:55:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:14 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:14 smithi119 ceph-mon[33714]: osdmap e702: 8 total, 8 up, 8 in 2024-03-24T11:55:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:55:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:55:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:15.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[30312]: osdmap e703: 8 total, 8 up, 8 in 2024-03-24T11:55:15.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:15.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:15.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:15.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[30312]: osdmap e704: 8 total, 8 up, 8 in 2024-03-24T11:55:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[42466]: osdmap e703: 8 total, 8 up, 8 in 2024-03-24T11:55:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:15 smithi114 ceph-mon[42466]: osdmap e704: 8 total, 8 up, 8 in 2024-03-24T11:55:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:15 smithi119 ceph-mon[33714]: osdmap e703: 8 total, 8 up, 8 in 2024-03-24T11:55:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:15 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:15 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:15.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:15 smithi119 ceph-mon[33714]: osdmap e704: 8 total, 8 up, 8 in 2024-03-24T11:55:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[42466]: pgmap v1094: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:55:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[42466]: osdmap e705: 8 total, 8 up, 8 in 2024-03-24T11:55:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[30312]: pgmap v1094: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:55:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:16.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:16 smithi114 ceph-mon[30312]: osdmap e705: 8 total, 8 up, 8 in 2024-03-24T11:55:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:16 smithi119 ceph-mon[33714]: pgmap v1094: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:55:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:16 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:16 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:16 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:16.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:16 smithi119 ceph-mon[33714]: osdmap e705: 8 total, 8 up, 8 in 2024-03-24T11:55:17.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]': finished 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[42466]: osdmap e706: 8 total, 8 up, 8 in 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]: dispatch 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]: dispatch 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]': finished 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[30312]: osdmap e706: 8 total, 8 up, 8 in 2024-03-24T11:55:17.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]: dispatch 2024-03-24T11:55:17.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]: dispatch 2024-03-24T11:55:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:17.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-142"}]': finished 2024-03-24T11:55:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:17 smithi119 ceph-mon[33714]: osdmap e706: 8 total, 8 up, 8 in 2024-03-24T11:55:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]: dispatch 2024-03-24T11:55:17.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]: dispatch 2024-03-24T11:55:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[42466]: pgmap v1097: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:55:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]': finished 2024-03-24T11:55:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[42466]: osdmap e707: 8 total, 8 up, 8 in 2024-03-24T11:55:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[30312]: pgmap v1097: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:55:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]': finished 2024-03-24T11:55:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[30312]: osdmap e707: 8 total, 8 up, 8 in 2024-03-24T11:55:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:18.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:18 smithi119 ceph-mon[33714]: pgmap v1097: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:55:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:18 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-142", "mode": "writeback"}]': finished 2024-03-24T11:55:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:18 smithi119 ceph-mon[33714]: osdmap e707: 8 total, 8 up, 8 in 2024-03-24T11:55:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:18.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:20.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:20.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:20.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[30312]: pgmap v1100: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:55:20.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[30312]: osdmap e708: 8 total, 8 up, 8 in 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[42466]: pgmap v1100: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[42466]: osdmap e708: 8 total, 8 up, 8 in 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:20.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:19 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:20.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:19 smithi119 ceph-mon[33714]: pgmap v1100: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-24T11:55:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:19 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:55:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:19 smithi119 ceph-mon[33714]: osdmap e708: 8 total, 8 up, 8 in 2024-03-24T11:55:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:19 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:20.304 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:19 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:21.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:55:21.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[30312]: osdmap e709: 8 total, 8 up, 8 in 2024-03-24T11:55:21.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:21.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:55:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[42466]: osdmap e709: 8 total, 8 up, 8 in 2024-03-24T11:55:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:21.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:21 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:21.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:21 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:55:21.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:21 smithi119 ceph-mon[33714]: osdmap e709: 8 total, 8 up, 8 in 2024-03-24T11:55:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:21.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:21 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[30312]: pgmap v1103: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:55:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[30312]: osdmap e710: 8 total, 8 up, 8 in 2024-03-24T11:55:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[42466]: pgmap v1103: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:55:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[42466]: osdmap e710: 8 total, 8 up, 8 in 2024-03-24T11:55:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:22.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:22 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:22.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:55:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:55:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:22 smithi119 ceph-mon[33714]: pgmap v1103: 332 pgs: 332 active+clean; 4.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:22 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:22 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:55:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:22 smithi119 ceph-mon[33714]: osdmap e710: 8 total, 8 up, 8 in 2024-03-24T11:55:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:22 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:22.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:22 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:23.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:55:23.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[42466]: osdmap e711: 8 total, 8 up, 8 in 2024-03-24T11:55:23.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:55:23.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:55:23.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:23.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:55:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[30312]: osdmap e711: 8 total, 8 up, 8 in 2024-03-24T11:55:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:55:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:55:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:23 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:23 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:55:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:23 smithi119 ceph-mon[33714]: osdmap e711: 8 total, 8 up, 8 in 2024-03-24T11:55:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:55:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:23 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-24T11:55:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:23.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:23 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[42466]: pgmap v1106: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-24T11:55:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[42466]: osdmap e712: 8 total, 8 up, 8 in 2024-03-24T11:55:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:55:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:55:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[30312]: pgmap v1106: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-24T11:55:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[30312]: osdmap e712: 8 total, 8 up, 8 in 2024-03-24T11:55:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:55:24.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:55:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:24 smithi119 ceph-mon[33714]: pgmap v1106: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-24T11:55:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:24 smithi119 ceph-mon[33714]: osdmap e712: 8 total, 8 up, 8 in 2024-03-24T11:55:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:55:24.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-24T11:55:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:55:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:55:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:25.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-24T11:55:25.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[42466]: osdmap e713: 8 total, 8 up, 8 in 2024-03-24T11:55:25.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:25.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-24T11:55:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[30312]: osdmap e713: 8 total, 8 up, 8 in 2024-03-24T11:55:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:25.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:25 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:25.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:25 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-24T11:55:25.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:25 smithi119 ceph-mon[33714]: osdmap e713: 8 total, 8 up, 8 in 2024-03-24T11:55:25.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:25.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:25 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[42466]: pgmap v1109: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:55:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[42466]: osdmap e714: 8 total, 8 up, 8 in 2024-03-24T11:55:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[30312]: pgmap v1109: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:55:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[30312]: osdmap e714: 8 total, 8 up, 8 in 2024-03-24T11:55:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:26.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:26 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:26 smithi119 ceph-mon[33714]: pgmap v1109: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:26 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:55:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:26 smithi119 ceph-mon[33714]: osdmap e714: 8 total, 8 up, 8 in 2024-03-24T11:55:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:26 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:26.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:26 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:27.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]': finished 2024-03-24T11:55:27.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[42466]: osdmap e715: 8 total, 8 up, 8 in 2024-03-24T11:55:27.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:27.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:27.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:27.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]': finished 2024-03-24T11:55:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[30312]: osdmap e715: 8 total, 8 up, 8 in 2024-03-24T11:55:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:27.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:27 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:27.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]': finished 2024-03-24T11:55:27.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:27 smithi119 ceph-mon[33714]: osdmap e715: 8 total, 8 up, 8 in 2024-03-24T11:55:27.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:27 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:27.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:27 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-142"}]: dispatch 2024-03-24T11:55:28.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:28 smithi114 ceph-mon[42466]: pgmap v1112: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:55:28.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:28 smithi114 ceph-mon[42466]: osdmap e716: 8 total, 8 up, 8 in 2024-03-24T11:55:28.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:28 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:28.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:28 smithi114 ceph-mon[30312]: pgmap v1112: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:55:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:28 smithi114 ceph-mon[30312]: osdmap e716: 8 total, 8 up, 8 in 2024-03-24T11:55:28.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:28 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:28 smithi119 ceph-mon[33714]: pgmap v1112: 332 pgs: 332 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-24T11:55:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:28 smithi119 ceph-mon[33714]: osdmap e716: 8 total, 8 up, 8 in 2024-03-24T11:55:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:28 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:29.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:29 smithi114 ceph-mon[42466]: osdmap e717: 8 total, 8 up, 8 in 2024-03-24T11:55:29.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:29 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:29.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:29 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:29.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:29 smithi114 ceph-mon[30312]: osdmap e717: 8 total, 8 up, 8 in 2024-03-24T11:55:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:29 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:29.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:29 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:29.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:29 smithi119 ceph-mon[33714]: osdmap e717: 8 total, 8 up, 8 in 2024-03-24T11:55:29.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:29 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:29.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:29 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:30.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[42466]: pgmap v1115: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:55:30.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:30.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[42466]: osdmap e718: 8 total, 8 up, 8 in 2024-03-24T11:55:30.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:30.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[30312]: pgmap v1115: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:55:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[30312]: osdmap e718: 8 total, 8 up, 8 in 2024-03-24T11:55:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:30.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:30 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:30 smithi119 ceph-mon[33714]: pgmap v1115: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:55:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:30 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:30 smithi119 ceph-mon[33714]: osdmap e718: 8 total, 8 up, 8 in 2024-03-24T11:55:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:30.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:30 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:31.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:31.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[42466]: osdmap e719: 8 total, 8 up, 8 in 2024-03-24T11:55:31.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:31.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[30312]: osdmap e719: 8 total, 8 up, 8 in 2024-03-24T11:55:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:31.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:31 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:31.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:31.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:31 smithi119 ceph-mon[33714]: osdmap e719: 8 total, 8 up, 8 in 2024-03-24T11:55:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:31.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:31 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:32.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[42466]: pgmap v1118: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:32.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]': finished 2024-03-24T11:55:32.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[42466]: osdmap e720: 8 total, 8 up, 8 in 2024-03-24T11:55:32.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]: dispatch 2024-03-24T11:55:32.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]: dispatch 2024-03-24T11:55:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[30312]: pgmap v1118: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]': finished 2024-03-24T11:55:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[30312]: osdmap e720: 8 total, 8 up, 8 in 2024-03-24T11:55:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]: dispatch 2024-03-24T11:55:32.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:32 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]: dispatch 2024-03-24T11:55:32.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:32 smithi119 ceph-mon[33714]: pgmap v1118: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:32 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-144"}]': finished 2024-03-24T11:55:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:32 smithi119 ceph-mon[33714]: osdmap e720: 8 total, 8 up, 8 in 2024-03-24T11:55:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]: dispatch 2024-03-24T11:55:32.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:32 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]: dispatch 2024-03-24T11:55:32.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:55:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:55:33.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:33.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:33.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]': finished 2024-03-24T11:55:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[42466]: osdmap e721: 8 total, 8 up, 8 in 2024-03-24T11:55:33.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]': finished 2024-03-24T11:55:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[30312]: osdmap e721: 8 total, 8 up, 8 in 2024-03-24T11:55:33.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:33 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:33.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:33 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:33 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-144", "mode": "readproxy"}]': finished 2024-03-24T11:55:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:33 smithi119 ceph-mon[33714]: osdmap e721: 8 total, 8 up, 8 in 2024-03-24T11:55:33.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:33 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:34.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:34 smithi119 ceph-mon[33714]: pgmap v1121: 332 pgs: 332 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:55:34.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:34.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:34 smithi114 ceph-mon[42466]: pgmap v1121: 332 pgs: 332 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:55:34.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:34 smithi114 ceph-mon[30312]: pgmap v1121: 332 pgs: 332 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T11:55:34.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:55:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:55:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:36.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:36 smithi119 ceph-mon[33714]: pgmap v1122: 332 pgs: 332 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 177 B/s wr, 1 op/s 2024-03-24T11:55:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:36 smithi114 ceph-mon[42466]: pgmap v1122: 332 pgs: 332 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 177 B/s wr, 1 op/s 2024-03-24T11:55:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:36 smithi114 ceph-mon[30312]: pgmap v1122: 332 pgs: 332 active+clean; 4.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 177 B/s wr, 1 op/s 2024-03-24T11:55:38.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:38 smithi119 ceph-mon[33714]: pgmap v1123: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 151 B/s wr, 1 op/s 2024-03-24T11:55:38.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:38 smithi114 ceph-mon[42466]: pgmap v1123: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 151 B/s wr, 1 op/s 2024-03-24T11:55:38.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:38 smithi114 ceph-mon[30312]: pgmap v1123: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 151 B/s wr, 1 op/s 2024-03-24T11:55:40.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:40 smithi119 ceph-mon[33714]: pgmap v1124: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-24T11:55:40.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:40 smithi114 ceph-mon[30312]: pgmap v1124: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-24T11:55:40.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:40 smithi114 ceph-mon[42466]: pgmap v1124: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-24T11:55:42.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:42 smithi119 ceph-mon[33714]: pgmap v1125: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 104 B/s wr, 1 op/s 2024-03-24T11:55:42.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:55:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:55:42.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:42 smithi114 ceph-mon[30312]: pgmap v1125: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 104 B/s wr, 1 op/s 2024-03-24T11:55:42.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:42 smithi114 ceph-mon[42466]: pgmap v1125: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 104 B/s wr, 1 op/s 2024-03-24T11:55:43.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:43.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:43.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:43 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:43.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:43 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:55:43.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:43.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:43.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:43.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:55:43.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:43.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:43.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:43.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:43 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:55:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: pgmap v1126: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:55:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:55:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: osdmap e722: 8 total, 8 up, 8 in 2024-03-24T11:55:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:44.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:44.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:44.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: pgmap v1126: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:55:44.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:55:44.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: osdmap e722: 8 total, 8 up, 8 in 2024-03-24T11:55:44.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: pgmap v1126: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:55:44.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: osdmap e722: 8 total, 8 up, 8 in 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:55:44.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:55:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:55:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:45.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:45.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]': finished 2024-03-24T11:55:45.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-mon[33714]: osdmap e723: 8 total, 8 up, 8 in 2024-03-24T11:55:45.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:45.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:45.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:45.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]': finished 2024-03-24T11:55:45.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[42466]: osdmap e723: 8 total, 8 up, 8 in 2024-03-24T11:55:45.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]': finished 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[30312]: osdmap e723: 8 total, 8 up, 8 in 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:45.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:45 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-144"}]: dispatch 2024-03-24T11:55:46.053 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 11:55:45 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=sqlstore.transactions t=2024-03-24T11:55:45.711081006Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:55:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:46 smithi119 ceph-mon[33714]: pgmap v1129: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:55:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:46 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:46.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:46 smithi119 ceph-mon[33714]: osdmap e724: 8 total, 8 up, 8 in 2024-03-24T11:55:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:46 smithi114 ceph-mon[30312]: pgmap v1129: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:55:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:46 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:46 smithi114 ceph-mon[30312]: osdmap e724: 8 total, 8 up, 8 in 2024-03-24T11:55:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:46 smithi114 ceph-mon[42466]: pgmap v1129: 332 pgs: 332 active+clean; 4.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:55:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:46 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:46.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:46 smithi114 ceph-mon[42466]: osdmap e724: 8 total, 8 up, 8 in 2024-03-24T11:55:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:47 smithi119 ceph-mon[33714]: osdmap e725: 8 total, 8 up, 8 in 2024-03-24T11:55:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:47 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:47 smithi114 ceph-mon[30312]: osdmap e725: 8 total, 8 up, 8 in 2024-03-24T11:55:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:47 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:47 smithi114 ceph-mon[42466]: osdmap e725: 8 total, 8 up, 8 in 2024-03-24T11:55:47.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:47.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:47 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:55:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:48 smithi119 ceph-mon[33714]: pgmap v1132: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:48 smithi119 ceph-mon[33714]: osdmap e726: 8 total, 8 up, 8 in 2024-03-24T11:55:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:48 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:48.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[30312]: pgmap v1132: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[30312]: osdmap e726: 8 total, 8 up, 8 in 2024-03-24T11:55:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:48.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:48.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[42466]: pgmap v1132: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:48.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:55:48.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[42466]: osdmap e726: 8 total, 8 up, 8 in 2024-03-24T11:55:48.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:48.741 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:48 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:55:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[30312]: osdmap e727: 8 total, 8 up, 8 in 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[42466]: osdmap e727: 8 total, 8 up, 8 in 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:49 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:55:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:49 smithi119 ceph-mon[33714]: osdmap e727: 8 total, 8 up, 8 in 2024-03-24T11:55:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:55:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:49 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:55:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:55:49.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:55:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: pgmap v1135: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]': finished 2024-03-24T11:55:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: osdmap e728: 8 total, 8 up, 8 in 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]: dispatch 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]: dispatch 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]': finished 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[30312]: osdmap e729: 8 total, 8 up, 8 in 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: pgmap v1135: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]': finished 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: osdmap e728: 8 total, 8 up, 8 in 2024-03-24T11:55:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]: dispatch 2024-03-24T11:55:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]: dispatch 2024-03-24T11:55:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]': finished 2024-03-24T11:55:50.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:50 smithi114 ceph-mon[42466]: osdmap e729: 8 total, 8 up, 8 in 2024-03-24T11:55:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: pgmap v1135: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:55:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi114-82677-111", "overlaypool": "test-rados-api-smithi114-82677-146"}]': finished 2024-03-24T11:55:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: osdmap e728: 8 total, 8 up, 8 in 2024-03-24T11:55:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]: dispatch 2024-03-24T11:55:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]: dispatch 2024-03-24T11:55:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-24T11:55:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi114-82677-146", "mode": "writeback"}]': finished 2024-03-24T11:55:50.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:50 smithi119 ceph-mon[33714]: osdmap e729: 8 total, 8 up, 8 in 2024-03-24T11:55:51.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:51.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:51 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:51.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:51.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:51 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:51.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:51.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:51 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-24T11:55:52.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:52 smithi119 ceph-mon[33714]: pgmap v1138: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:52 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:55:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:52 smithi119 ceph-mon[33714]: osdmap e730: 8 total, 8 up, 8 in 2024-03-24T11:55:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:52.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:52 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:52.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:55:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:55:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[42466]: pgmap v1138: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:55:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[42466]: osdmap e730: 8 total, 8 up, 8 in 2024-03-24T11:55:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[30312]: pgmap v1138: 332 pgs: 32 creating+peering, 300 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:55:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_count","val": "2"}]': finished 2024-03-24T11:55:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[30312]: osdmap e730: 8 total, 8 up, 8 in 2024-03-24T11:55:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:52.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:52 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-24T11:55:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:55:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: osdmap e731: 8 total, 8 up, 8 in 2024-03-24T11:55:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: osdmap e732: 8 total, 8 up, 8 in 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: osdmap e731: 8 total, 8 up, 8 in 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:53.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:55:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: osdmap e732: 8 total, 8 up, 8 in 2024-03-24T11:55:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:53.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:53 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:53.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:55:53.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_period","val": "600"}]': finished 2024-03-24T11:55:53.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: osdmap e731: 8 total, 8 up, 8 in 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: osdmap e732: 8 total, 8 up, 8 in 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:53.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:53 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[30312]: pgmap v1141: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[30312]: osdmap e733: 8 total, 8 up, 8 in 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[42466]: pgmap v1141: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:55:54.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[42466]: osdmap e733: 8 total, 8 up, 8 in 2024-03-24T11:55:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:55:54.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:55:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:54 smithi119 ceph-mon[33714]: pgmap v1141: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:55:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:54 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-24T11:55:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:54 smithi119 ceph-mon[33714]: osdmap e733: 8 total, 8 up, 8 in 2024-03-24T11:55:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:54 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:55:54.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:54 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-24T11:55:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:55:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:55:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:55:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:56 smithi114 ceph-mon[30312]: pgmap v1144: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:55:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:56 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]': finished 2024-03-24T11:55:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:56 smithi114 ceph-mon[30312]: osdmap e734: 8 total, 8 up, 8 in 2024-03-24T11:55:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:56 smithi114 ceph-mon[42466]: pgmap v1144: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:55:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:56 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]': finished 2024-03-24T11:55:56.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:56 smithi114 ceph-mon[42466]: osdmap e734: 8 total, 8 up, 8 in 2024-03-24T11:55:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:56 smithi119 ceph-mon[33714]: pgmap v1144: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-24T11:55:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:56 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-146","var": "target_max_objects","val": "1"}]': finished 2024-03-24T11:55:56.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:56 smithi119 ceph-mon[33714]: osdmap e734: 8 total, 8 up, 8 in 2024-03-24T11:55:57.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:57 smithi114 ceph-mon[30312]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-24T11:55:57.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:57 smithi114 ceph-mon[42466]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-24T11:55:57.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:57 smithi119 ceph-mon[33714]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-24T11:55:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:55:58 smithi114 ceph-mon[30312]: pgmap v1146: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:55:58.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:55:58 smithi114 ceph-mon[42466]: pgmap v1146: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:55:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:55:58 smithi119 ceph-mon[33714]: pgmap v1146: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:00 smithi114 ceph-mon[30312]: pgmap v1147: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:00.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:00 smithi114 ceph-mon[42466]: pgmap v1147: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:00 smithi119 ceph-mon[33714]: pgmap v1147: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:02.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:02 smithi119 ceph-mon[33714]: pgmap v1148: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 666 B/s rd, 0 B/s wr, 0 op/s 2024-03-24T11:56:02.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:56:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:56:02.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:02 smithi114 ceph-mon[30312]: pgmap v1148: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 666 B/s rd, 0 B/s wr, 0 op/s 2024-03-24T11:56:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:02 smithi114 ceph-mon[42466]: pgmap v1148: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 666 B/s rd, 0 B/s wr, 0 op/s 2024-03-24T11:56:03.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:03.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:03.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:04 smithi114 ceph-mon[42466]: pgmap v1149: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:04 smithi114 ceph-mon[30312]: pgmap v1149: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:04 smithi119 ceph-mon[33714]: pgmap v1149: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:56:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:56:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:05.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:05.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:05 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:05.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:05.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:05 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:05.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:05.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:05 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[30312]: pgmap v1150: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[30312]: osdmap e735: 8 total, 8 up, 8 in 2024-03-24T11:56:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[42466]: pgmap v1150: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[42466]: osdmap e735: 8 total, 8 up, 8 in 2024-03-24T11:56:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:06.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:06 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:06 smithi119 ceph-mon[33714]: pgmap v1150: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:06 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:06 smithi119 ceph-mon[33714]: osdmap e735: 8 total, 8 up, 8 in 2024-03-24T11:56:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:06 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:07.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]': finished 2024-03-24T11:56:07.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[30312]: osdmap e736: 8 total, 8 up, 8 in 2024-03-24T11:56:07.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:07.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]': finished 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[42466]: osdmap e736: 8 total, 8 up, 8 in 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:07.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:07 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:07.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:07 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]': finished 2024-03-24T11:56:07.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:07 smithi119 ceph-mon[33714]: osdmap e736: 8 total, 8 up, 8 in 2024-03-24T11:56:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:07 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:07 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:07.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:07 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-146"}]: dispatch 2024-03-24T11:56:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[30312]: pgmap v1153: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[30312]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-24T11:56:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[30312]: osdmap e737: 8 total, 8 up, 8 in 2024-03-24T11:56:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[42466]: pgmap v1153: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[42466]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-24T11:56:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:08 smithi114 ceph-mon[42466]: osdmap e737: 8 total, 8 up, 8 in 2024-03-24T11:56:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:08 smithi119 ceph-mon[33714]: pgmap v1153: 332 pgs: 332 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-24T11:56:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:08 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:08 smithi119 ceph-mon[33714]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-24T11:56:08.805 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:08 smithi119 ceph-mon[33714]: osdmap e737: 8 total, 8 up, 8 in 2024-03-24T11:56:09.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[30312]: osdmap e738: 8 total, 8 up, 8 in 2024-03-24T11:56:09.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:09.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:09.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[30312]: osdmap e739: 8 total, 8 up, 8 in 2024-03-24T11:56:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[42466]: osdmap e738: 8 total, 8 up, 8 in 2024-03-24T11:56:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:09.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:09 smithi114 ceph-mon[42466]: osdmap e739: 8 total, 8 up, 8 in 2024-03-24T11:56:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:09 smithi119 ceph-mon[33714]: osdmap e738: 8 total, 8 up, 8 in 2024-03-24T11:56:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:09 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:09 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:09 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:09.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:09 smithi119 ceph-mon[33714]: osdmap e739: 8 total, 8 up, 8 in 2024-03-24T11:56:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:10 smithi114 ceph-mon[30312]: pgmap v1156: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:56:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:56:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:10 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:56:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:10 smithi114 ceph-mon[42466]: pgmap v1156: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:56:10.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:56:10.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:10 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:56:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:10 smithi119 ceph-mon[33714]: pgmap v1156: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-24T11:56:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:56:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:10 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-24T11:56:11.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:56:11.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[30312]: osdmap e740: 8 total, 8 up, 8 in 2024-03-24T11:56:11.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:11.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:56:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[42466]: osdmap e740: 8 total, 8 up, 8 in 2024-03-24T11:56:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:11.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:11 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:11.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:11 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-24T11:56:11.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:11 smithi119 ceph-mon[33714]: osdmap e740: 8 total, 8 up, 8 in 2024-03-24T11:56:11.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:11 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:11.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:11 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-mon[33714]: pgmap v1159: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]': finished 2024-03-24T11:56:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-mon[33714]: osdmap e741: 8 total, 8 up, 8 in 2024-03-24T11:56:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:12.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:12.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:56:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:56:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[30312]: pgmap v1159: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]': finished 2024-03-24T11:56:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[30312]: osdmap e741: 8 total, 8 up, 8 in 2024-03-24T11:56:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[42466]: pgmap v1159: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]': finished 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[42466]: osdmap e741: 8 total, 8 up, 8 in 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:12.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:12 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-148"}]: dispatch 2024-03-24T11:56:13.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:13.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:13 smithi114 ceph-mon[30312]: osdmap e742: 8 total, 8 up, 8 in 2024-03-24T11:56:13.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:13 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:13.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:13.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:13 smithi114 ceph-mon[42466]: osdmap e742: 8 total, 8 up, 8 in 2024-03-24T11:56:13.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:13 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:13.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:13.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:13 smithi119 ceph-mon[33714]: osdmap e742: 8 total, 8 up, 8 in 2024-03-24T11:56:13.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:13 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:14 smithi119 ceph-mon[33714]: pgmap v1162: 300 pgs: 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:56:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:14 smithi119 ceph-mon[33714]: osdmap e743: 8 total, 8 up, 8 in 2024-03-24T11:56:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:14 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:14 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:56:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[30312]: pgmap v1162: 300 pgs: 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:56:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[30312]: osdmap e743: 8 total, 8 up, 8 in 2024-03-24T11:56:14.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:14.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:14.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[42466]: pgmap v1162: 300 pgs: 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:56:14.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[42466]: osdmap e743: 8 total, 8 up, 8 in 2024-03-24T11:56:14.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:14.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:14 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:15.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:15 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:15.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:15 smithi119 ceph-mon[33714]: osdmap e744: 8 total, 8 up, 8 in 2024-03-24T11:56:15.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:15 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:15 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-150"}]: dispatch 2024-03-24T11:56:15.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:15 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-150"}]: dispatch 2024-03-24T11:56:15.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:15.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[30312]: osdmap e744: 8 total, 8 up, 8 in 2024-03-24T11:56:15.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:15.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:15.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-150"}]: dispatch 2024-03-24T11:56:15.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-150"}]: dispatch 2024-03-24T11:56:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[42466]: osdmap e744: 8 total, 8 up, 8 in 2024-03-24T11:56:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-150"}]: dispatch 2024-03-24T11:56:15.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:15 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-150"}]: dispatch 2024-03-24T11:56:16.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:16 smithi119 ceph-mon[33714]: pgmap v1165: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:56:16.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:16 smithi119 ceph-mon[33714]: osdmap e745: 8 total, 8 up, 8 in 2024-03-24T11:56:16.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:16 smithi114 ceph-mon[30312]: pgmap v1165: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:56:16.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:16 smithi114 ceph-mon[30312]: osdmap e745: 8 total, 8 up, 8 in 2024-03-24T11:56:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:16 smithi114 ceph-mon[42466]: pgmap v1165: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:56:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:16 smithi114 ceph-mon[42466]: osdmap e745: 8 total, 8 up, 8 in 2024-03-24T11:56:17.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:17 smithi119 ceph-mon[33714]: osdmap e746: 8 total, 8 up, 8 in 2024-03-24T11:56:17.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:17 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:17.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:17 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:17.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:17 smithi114 ceph-mon[30312]: osdmap e746: 8 total, 8 up, 8 in 2024-03-24T11:56:17.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:17 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:17.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:17 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:17.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:17 smithi114 ceph-mon[42466]: osdmap e746: 8 total, 8 up, 8 in 2024-03-24T11:56:17.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:17 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:17.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:17 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: pgmap v1168: 332 pgs: 29 creating+peering, 3 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:56:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: osdmap e747: 8 total, 8 up, 8 in 2024-03-24T11:56:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-152"}]: dispatch 2024-03-24T11:56:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-152"}]: dispatch 2024-03-24T11:56:18.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:18 smithi119 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: pgmap v1168: 332 pgs: 29 creating+peering, 3 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: osdmap e747: 8 total, 8 up, 8 in 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-152"}]: dispatch 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-152"}]: dispatch 2024-03-24T11:56:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[42466]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: pgmap v1168: 332 pgs: 29 creating+peering, 3 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: osdmap e747: 8 total, 8 up, 8 in 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-152"}]: dispatch 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-152"}]: dispatch 2024-03-24T11:56:18.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:18 smithi114 ceph-mon[30312]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:19.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:19 smithi119 ceph-mon[33714]: osdmap e748: 8 total, 8 up, 8 in 2024-03-24T11:56:19.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:19.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:19.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:19 smithi114 ceph-mon[30312]: osdmap e748: 8 total, 8 up, 8 in 2024-03-24T11:56:19.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:19.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:19 smithi114 ceph-mon[42466]: osdmap e748: 8 total, 8 up, 8 in 2024-03-24T11:56:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:20 smithi119 ceph-mon[33714]: pgmap v1171: 300 pgs: 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:56:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:20 smithi119 ceph-mon[33714]: osdmap e749: 8 total, 8 up, 8 in 2024-03-24T11:56:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:20 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:20.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:20 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[30312]: pgmap v1171: 300 pgs: 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:56:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[30312]: osdmap e749: 8 total, 8 up, 8 in 2024-03-24T11:56:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:20.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[42466]: pgmap v1171: 300 pgs: 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:56:20.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[42466]: osdmap e749: 8 total, 8 up, 8 in 2024-03-24T11:56:20.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:20.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:20 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:56:21.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:21.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: osdmap e750: 8 total, 8 up, 8 in 2024-03-24T11:56:21.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-111","var": "dedup_tier","val": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-111","var": "dedup_tier","val": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:21 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: osdmap e750: 8 total, 8 up, 8 in 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-111","var": "dedup_tier","val": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-111","var": "dedup_tier","val": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-82677-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: osdmap e750: 8 total, 8 up, 8 in 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-111","var": "dedup_tier","val": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi114-82677-111","var": "dedup_tier","val": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:21.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:21 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi114-82677-111", "tierpool": "test-rados-api-smithi114-82677-154"}]: dispatch 2024-03-24T11:56:22.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:56:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:56:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:22 smithi114 ceph-mon[42466]: pgmap v1174: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:22 smithi114 ceph-mon[42466]: osdmap e751: 8 total, 8 up, 8 in 2024-03-24T11:56:22.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:22 smithi114 ceph-mon[30312]: pgmap v1174: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:22.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:22 smithi114 ceph-mon[30312]: osdmap e751: 8 total, 8 up, 8 in 2024-03-24T11:56:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:22 smithi119 ceph-mon[33714]: pgmap v1174: 332 pgs: 32 unknown, 300 active+clean; 4.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:22 smithi119 ceph-mon[33714]: osdmap e751: 8 total, 8 up, 8 in 2024-03-24T11:56:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[30312]: osdmap e752: 8 total, 8 up, 8 in 2024-03-24T11:56:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[30312]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:23.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:23.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[42466]: osdmap e752: 8 total, 8 up, 8 in 2024-03-24T11:56:23.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:23.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:23.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:23 smithi114 ceph-mon[42466]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:23 smithi119 ceph-mon[33714]: osdmap e752: 8 total, 8 up, 8 in 2024-03-24T11:56:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:23 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:23 smithi119 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:24.581 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7873 ms) 2024-03-24T11:56:24.581 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-24T11:56:24.581 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12174 ms) 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8069 ms) 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12728 ms) 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8051 ms) 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8054 ms) 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: ok, hit_set contains 307:602f83fe:::foo:head 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8119 ms) 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,0 2024-03-24T11:56:24.582 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: first is 1711281300 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,1711281305,1711281306,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,1711281305,1711281306,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,1711281305,1711281306,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,1711281305,1711281306,1711281308,1711281309,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,1711281305,1711281306,1711281308,1711281309,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281300,1711281302,1711281303,1711281305,1711281306,1711281308,1711281309,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: got ls 1711281303,1711281305,1711281306,1711281308,1711281309,1711281311,1711281312,0 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: first now 1711281303, trimmed 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21080 ms) 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-24T11:56:24.583 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: foo0 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14072 ms) 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18076 ms) 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22157 ms) 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5035 ms) 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3032 ms) 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3011 ms) 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-24T11:56:24.584 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3032 ms) 2024-03-24T11:56:24.585 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (230075 ms total) 2024-03-24T11:56:24.585 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: 2024-03-24T11:56:24.585 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-24T11:56:24.585 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (856638 ms total) 2024-03-24T11:56:24.585 INFO:tasks.workunit.client.0.smithi114.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82382 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82382 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82602 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82602 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83093 2024-03-24T11:56:24.587 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83093 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82792 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82792 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82740 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82740 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82465 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82465 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82767 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82767 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83291 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83291 2024-03-24T11:56:24.588 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.589 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83331 2024-03-24T11:56:24.589 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83331 2024-03-24T11:56:24.589 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.589 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82484 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82484 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83163 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83163 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83002 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83002 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82357 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82357 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82417 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82417 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.590 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83035 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83035 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82347 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82347 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82368 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82368 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83537 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83537 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82644 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82644 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83607 2024-03-24T11:56:24.591 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83607 2024-03-24T11:56:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:56:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[30312]: pgmap v1177: 292 pgs: 292 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[30312]: osdmap e753: 8 total, 8 up, 8 in 2024-03-24T11:56:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:24.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[42466]: pgmap v1177: 292 pgs: 292 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[42466]: osdmap e753: 8 total, 8 up, 8 in 2024-03-24T11:56:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:24.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:24 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:24 smithi119 ceph-mon[33714]: pgmap v1177: 292 pgs: 292 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:24 smithi119 ceph-mon[33714]: osdmap e753: 8 total, 8 up, 8 in 2024-03-24T11:56:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3093589231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:24 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]: dispatch 2024-03-24T11:56:25.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:25 smithi114 ceph-mon[30312]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:25.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:25 smithi114 ceph-mon[30312]: osdmap e754: 8 total, 8 up, 8 in 2024-03-24T11:56:25.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:25 smithi114 ceph-mon[42466]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:25.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:25 smithi114 ceph-mon[42466]: osdmap e754: 8 total, 8 up, 8 in 2024-03-24T11:56:26.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:25 smithi119 ceph-mon[33714]: from='client.56069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi114-82677-111"}]': finished 2024-03-24T11:56:26.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:25 smithi119 ceph-mon[33714]: osdmap e754: 8 total, 8 up, 8 in 2024-03-24T11:56:26.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:26 smithi114 ceph-mon[30312]: pgmap v1180: 292 pgs: 292 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:26 smithi114 ceph-mon[42466]: pgmap v1180: 292 pgs: 292 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:27.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:26 smithi119 ceph-mon[33714]: pgmap v1180: 292 pgs: 292 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:28.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:28 smithi114 ceph-mon[30312]: pgmap v1181: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:28.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:28 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:28 smithi114 ceph-mon[42466]: pgmap v1181: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:28 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:28 smithi119 ceph-mon[33714]: pgmap v1181: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:56:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:28 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:56:30.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:30 smithi114 ceph-mon[30312]: pgmap v1182: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-03-24T11:56:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:30 smithi114 ceph-mon[42466]: pgmap v1182: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-03-24T11:56:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:30 smithi119 ceph-mon[33714]: pgmap v1182: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-03-24T11:56:32.555 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:56:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:56:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:32 smithi114 ceph-mon[30312]: pgmap v1183: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:56:32.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:32 smithi114 ceph-mon[42466]: pgmap v1183: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:56:33.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:32 smithi119 ceph-mon[33714]: pgmap v1183: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:56:33.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:33.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:34.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:34 smithi114 ceph-mon[30312]: pgmap v1184: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:56:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:56:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:56:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:34 smithi114 ceph-mon[42466]: pgmap v1184: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:56:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:34 smithi119 ceph-mon[33714]: pgmap v1184: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:56:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:36.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:36 smithi114 ceph-mon[30312]: pgmap v1185: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-24T11:56:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:36 smithi114 ceph-mon[42466]: pgmap v1185: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-24T11:56:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:36 smithi119 ceph-mon[33714]: pgmap v1185: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-24T11:56:38.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:38 smithi114 ceph-mon[30312]: pgmap v1186: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:38.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:38 smithi114 ceph-mon[42466]: pgmap v1186: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:39.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:38 smithi119 ceph-mon[33714]: pgmap v1186: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:40 smithi114 ceph-mon[30312]: pgmap v1187: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:40 smithi114 ceph-mon[42466]: pgmap v1187: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:41.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:40 smithi119 ceph-mon[33714]: pgmap v1187: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:42.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:56:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:56:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:42 smithi114 ceph-mon[30312]: pgmap v1188: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:42 smithi114 ceph-mon[42466]: pgmap v1188: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:43.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:42 smithi119 ceph-mon[33714]: pgmap v1188: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:43.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:44 smithi114 ceph-mon[30312]: pgmap v1189: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:44 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:56:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:56:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:44 smithi114 ceph-mon[42466]: pgmap v1189: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:44 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:44 smithi119 ceph-mon[33714]: pgmap v1189: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:44 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[30312]: pgmap v1190: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:45.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:45 smithi114 ceph-mon[42466]: pgmap v1190: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:46.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:56:46.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:56:46.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:56:46.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:45 smithi119 ceph-mon[33714]: pgmap v1190: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:48.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:48 smithi114 ceph-mon[42466]: pgmap v1191: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:48.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:48 smithi114 ceph-mon[30312]: pgmap v1191: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:48 smithi119 ceph-mon[33714]: pgmap v1191: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:49.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:49.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:49.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:56:50.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:50 smithi114 ceph-mon[30312]: pgmap v1192: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:50.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:50 smithi114 ceph-mon[42466]: pgmap v1192: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:50.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:50 smithi119 ceph-mon[33714]: pgmap v1192: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:52.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:52 smithi114 ceph-mon[30312]: pgmap v1193: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:56:52.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:52 smithi114 ceph-mon[42466]: pgmap v1193: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:56:52.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:52 smithi119 ceph-mon[33714]: pgmap v1193: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:56:52.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:56:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:56:53.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:53.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:53.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:56:54.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:54 smithi114 ceph-mon[42466]: pgmap v1194: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:54.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:54 smithi114 ceph-mon[30312]: pgmap v1194: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:54.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:54 smithi119 ceph-mon[33714]: pgmap v1194: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:56:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:56:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:56:56.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:56 smithi114 ceph-mon[30312]: pgmap v1195: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:56 smithi114 ceph-mon[42466]: pgmap v1195: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:56.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:56 smithi119 ceph-mon[33714]: pgmap v1195: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:56:58.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:56:58 smithi114 ceph-mon[30312]: pgmap v1196: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:58.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:56:58 smithi114 ceph-mon[42466]: pgmap v1196: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:56:58.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:56:58 smithi119 ceph-mon[33714]: pgmap v1196: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:00.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:00 smithi114 ceph-mon[30312]: pgmap v1197: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:00.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:00 smithi114 ceph-mon[42466]: pgmap v1197: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:00.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:00 smithi119 ceph-mon[33714]: pgmap v1197: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:02.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:02 smithi114 ceph-mon[30312]: pgmap v1198: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:02.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:02 smithi114 ceph-mon[42466]: pgmap v1198: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:02.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:02 smithi119 ceph-mon[33714]: pgmap v1198: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:02.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:57:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:57:03.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:03.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:03.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:04 smithi114 ceph-mon[30312]: pgmap v1199: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:04.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:04 smithi114 ceph-mon[42466]: pgmap v1199: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:04.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:04 smithi119 ceph-mon[33714]: pgmap v1199: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:57:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:57:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:06 smithi114 ceph-mon[30312]: pgmap v1200: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:06.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:06 smithi114 ceph-mon[42466]: pgmap v1200: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:06.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:06 smithi119 ceph-mon[33714]: pgmap v1200: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:08.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:08 smithi114 ceph-mon[30312]: pgmap v1201: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:08.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:08 smithi114 ceph-mon[42466]: pgmap v1201: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:08.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:08 smithi119 ceph-mon[33714]: pgmap v1201: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:10 smithi114 ceph-mon[42466]: pgmap v1202: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:10.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:10 smithi114 ceph-mon[30312]: pgmap v1202: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:10 smithi119 ceph-mon[33714]: pgmap v1202: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:12 smithi114 ceph-mon[30312]: pgmap v1203: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:12 smithi114 ceph-mon[42466]: pgmap v1203: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:12 smithi119 ceph-mon[33714]: pgmap v1203: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:12.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:57:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:57:13.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:14 smithi114 ceph-mon[30312]: pgmap v1204: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:14 smithi114 ceph-mon[42466]: pgmap v1204: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:14 smithi119 ceph-mon[33714]: pgmap v1204: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:57:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:57:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:16.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:16 smithi114 ceph-mon[30312]: pgmap v1205: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:16 smithi114 ceph-mon[42466]: pgmap v1205: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:16 smithi119 ceph-mon[33714]: pgmap v1205: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:18 smithi114 ceph-mon[30312]: pgmap v1206: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:18 smithi114 ceph-mon[42466]: pgmap v1206: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:18 smithi119 ceph-mon[33714]: pgmap v1206: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:19.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:19.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:19.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:20.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:20 smithi114 ceph-mon[30312]: pgmap v1207: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:20.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:20 smithi114 ceph-mon[42466]: pgmap v1207: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:20.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:20 smithi119 ceph-mon[33714]: pgmap v1207: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:22.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:22 smithi114 ceph-mon[30312]: pgmap v1208: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:22.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:22 smithi114 ceph-mon[42466]: pgmap v1208: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:22.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:22 smithi119 ceph-mon[33714]: pgmap v1208: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:22.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:57:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:57:23.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:23.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:24.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:24 smithi114 ceph-mon[30312]: pgmap v1209: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:24 smithi114 ceph-mon[42466]: pgmap v1209: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:24 smithi119 ceph-mon[33714]: pgmap v1209: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:57:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:57:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:26.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:26 smithi114 ceph-mon[30312]: pgmap v1210: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:26.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:26 smithi114 ceph-mon[42466]: pgmap v1210: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:26 smithi119 ceph-mon[33714]: pgmap v1210: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:28.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:28 smithi114 ceph-mon[30312]: pgmap v1211: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:28.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:28 smithi114 ceph-mon[42466]: pgmap v1211: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:28 smithi119 ceph-mon[33714]: pgmap v1211: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:30.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:30 smithi114 ceph-mon[30312]: pgmap v1212: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:30.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:30 smithi114 ceph-mon[42466]: pgmap v1212: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:30 smithi119 ceph-mon[33714]: pgmap v1212: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:32.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:32 smithi114 ceph-mon[30312]: pgmap v1213: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:32.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:32 smithi114 ceph-mon[42466]: pgmap v1213: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:32.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:32 smithi119 ceph-mon[33714]: pgmap v1213: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:32.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:57:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:57:33.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:33.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:33.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:34.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:34 smithi114 ceph-mon[42466]: pgmap v1214: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:34.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:34.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:34 smithi114 ceph-mon[30312]: pgmap v1214: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:34.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:34.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:34 smithi119 ceph-mon[33714]: pgmap v1214: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:34.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:57:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:57:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:36.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:36 smithi114 ceph-mon[30312]: pgmap v1215: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:36.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:36 smithi114 ceph-mon[42466]: pgmap v1215: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:36.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:36 smithi119 ceph-mon[33714]: pgmap v1215: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:38.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:38 smithi114 ceph-mon[30312]: pgmap v1216: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:38.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:38 smithi114 ceph-mon[42466]: pgmap v1216: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:38.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:38 smithi119 ceph-mon[33714]: pgmap v1216: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:40.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:40 smithi114 ceph-mon[30312]: pgmap v1217: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:40.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:40 smithi114 ceph-mon[42466]: pgmap v1217: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:40.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:40 smithi119 ceph-mon[33714]: pgmap v1217: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:42.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:42 smithi114 ceph-mon[30312]: pgmap v1218: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:42.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:42 smithi114 ceph-mon[42466]: pgmap v1218: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:42.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:42 smithi119 ceph-mon[33714]: pgmap v1218: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:57:42.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:57:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:57:43.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:43.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:43.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:44.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:44 smithi114 ceph-mon[30312]: pgmap v1219: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:44.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:44 smithi114 ceph-mon[42466]: pgmap v1219: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:44.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:44 smithi119 ceph-mon[33714]: pgmap v1219: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:57:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:57:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:45.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:45 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:45.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:45 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:45.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:45 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:57:46.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[30312]: pgmap v1220: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:46.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:46.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:46.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:57:46.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[42466]: pgmap v1220: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:46.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:46 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:57:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:46 smithi119 ceph-mon[33714]: pgmap v1220: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:46 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:57:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:46 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:57:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:46 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:57:48.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:48 smithi114 ceph-mon[30312]: pgmap v1221: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:48.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:48 smithi114 ceph-mon[42466]: pgmap v1221: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:48 smithi119 ceph-mon[33714]: pgmap v1221: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:49.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:49.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:49.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:57:50.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:50 smithi114 ceph-mon[30312]: pgmap v1222: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:50 smithi114 ceph-mon[42466]: pgmap v1222: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:50.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:50 smithi119 ceph-mon[33714]: pgmap v1222: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:52.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:52 smithi114 ceph-mon[30312]: pgmap v1223: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:52.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:52 smithi114 ceph-mon[42466]: pgmap v1223: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:52.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:52 smithi119 ceph-mon[33714]: pgmap v1223: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:52.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:57:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:57:53.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:53.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:53.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:57:54.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:54 smithi114 ceph-mon[30312]: pgmap v1224: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:54.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:54 smithi114 ceph-mon[42466]: pgmap v1224: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:54.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:54 smithi119 ceph-mon[33714]: pgmap v1224: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:57:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:57:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:57:56.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:56 smithi114 ceph-mon[30312]: pgmap v1225: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:56 smithi114 ceph-mon[42466]: pgmap v1225: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:56.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:56 smithi119 ceph-mon[33714]: pgmap v1225: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:57:58.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:57:58 smithi119 ceph-mon[33714]: pgmap v1226: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:57:58 smithi114 ceph-mon[30312]: pgmap v1226: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:57:58.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:57:58 smithi114 ceph-mon[42466]: pgmap v1226: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:00.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:00 smithi119 ceph-mon[33714]: pgmap v1227: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:00.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:00 smithi114 ceph-mon[30312]: pgmap v1227: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:00.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:00 smithi114 ceph-mon[42466]: pgmap v1227: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:02.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:02 smithi114 ceph-mon[30312]: pgmap v1228: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:02.540 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:02 smithi114 ceph-mon[42466]: pgmap v1228: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:02.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:02 smithi119 ceph-mon[33714]: pgmap v1228: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:02.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:58:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:58:03.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:03.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:03.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:04 smithi119 ceph-mon[33714]: pgmap v1229: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:04.653 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:04 smithi114 ceph-mon[30312]: pgmap v1229: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:04.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:04 smithi114 ceph-mon[42466]: pgmap v1229: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:04.654 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:58:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:58:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:06.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:06 smithi119 ceph-mon[33714]: pgmap v1230: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:06 smithi114 ceph-mon[30312]: pgmap v1230: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:06 smithi114 ceph-mon[42466]: pgmap v1230: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:08.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:08 smithi119 ceph-mon[33714]: pgmap v1231: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:08 smithi114 ceph-mon[30312]: pgmap v1231: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:08 smithi114 ceph-mon[42466]: pgmap v1231: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:10 smithi119 ceph-mon[33714]: pgmap v1232: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:10 smithi114 ceph-mon[30312]: pgmap v1232: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:10 smithi114 ceph-mon[42466]: pgmap v1232: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:12 smithi119 ceph-mon[33714]: pgmap v1233: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:12.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:58:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:58:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:12 smithi114 ceph-mon[30312]: pgmap v1233: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:12.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:12 smithi114 ceph-mon[42466]: pgmap v1233: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:13.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:13.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:14 smithi119 ceph-mon[33714]: pgmap v1234: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:14.654 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:14 smithi114 ceph-mon[30312]: pgmap v1234: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:14.655 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:14 smithi114 ceph-mon[42466]: pgmap v1234: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:58:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:58:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:16 smithi119 ceph-mon[33714]: pgmap v1235: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:16.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:16 smithi114 ceph-mon[30312]: pgmap v1235: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:16 smithi114 ceph-mon[42466]: pgmap v1235: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:18 smithi119 ceph-mon[33714]: pgmap v1236: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:18.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:18 smithi114 ceph-mon[30312]: pgmap v1236: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:18.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:18 smithi114 ceph-mon[42466]: pgmap v1236: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:19.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:19.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:19.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:20.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:20 smithi119 ceph-mon[33714]: pgmap v1237: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:20 smithi114 ceph-mon[30312]: pgmap v1237: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:20 smithi114 ceph-mon[42466]: pgmap v1237: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:22.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:22 smithi119 ceph-mon[33714]: pgmap v1238: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:22.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:58:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:58:22.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:22 smithi114 ceph-mon[30312]: pgmap v1238: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:22 smithi114 ceph-mon[42466]: pgmap v1238: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:23.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:23.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:24 smithi119 ceph-mon[33714]: pgmap v1239: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:24.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:24 smithi114 ceph-mon[30312]: pgmap v1239: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:24.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:24 smithi114 ceph-mon[42466]: pgmap v1239: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:58:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:58:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:26 smithi119 ceph-mon[33714]: pgmap v1240: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:26.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:26 smithi114 ceph-mon[30312]: pgmap v1240: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:26.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:26 smithi114 ceph-mon[42466]: pgmap v1240: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:28 smithi119 ceph-mon[33714]: pgmap v1241: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:28.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:28 smithi114 ceph-mon[30312]: pgmap v1241: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:28.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:28 smithi114 ceph-mon[42466]: pgmap v1241: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:30 smithi119 ceph-mon[33714]: pgmap v1242: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:30.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:30 smithi114 ceph-mon[30312]: pgmap v1242: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:30.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:30 smithi114 ceph-mon[42466]: pgmap v1242: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:32.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:32 smithi114 ceph-mon[30312]: pgmap v1243: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:32.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:32 smithi114 ceph-mon[42466]: pgmap v1243: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:32 smithi119 ceph-mon[33714]: pgmap v1243: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:32.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:58:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:58:33.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:33.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:33.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:34.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:34 smithi114 ceph-mon[42466]: pgmap v1244: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:34.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:34 smithi114 ceph-mon[30312]: pgmap v1244: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:34 smithi119 ceph-mon[33714]: pgmap v1244: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:58:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:58:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:36 smithi114 ceph-mon[30312]: pgmap v1245: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:36 smithi114 ceph-mon[42466]: pgmap v1245: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:36 smithi119 ceph-mon[33714]: pgmap v1245: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:38.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:38 smithi114 ceph-mon[30312]: pgmap v1246: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:38.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:38 smithi114 ceph-mon[42466]: pgmap v1246: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:38.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:38 smithi119 ceph-mon[33714]: pgmap v1246: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:40.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:40 smithi114 ceph-mon[42466]: pgmap v1247: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:40.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:40 smithi114 ceph-mon[30312]: pgmap v1247: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:40 smithi119 ceph-mon[33714]: pgmap v1247: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:42.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:42 smithi114 ceph-mon[30312]: pgmap v1248: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:42.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:42 smithi114 ceph-mon[42466]: pgmap v1248: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:42 smithi119 ceph-mon[33714]: pgmap v1248: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:42.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:58:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:58:43.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:43.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:43.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:44.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:44 smithi114 ceph-mon[30312]: pgmap v1249: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:44.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:44 smithi114 ceph-mon[42466]: pgmap v1249: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:44 smithi119 ceph-mon[33714]: pgmap v1249: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:58:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:58:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:46 smithi114 ceph-mon[30312]: pgmap v1250: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:46 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:46 smithi114 ceph-mon[42466]: pgmap v1250: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:46 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:46 smithi119 ceph-mon[33714]: pgmap v1250: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:46 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:58:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:47 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:47 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:47 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:58:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:47 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:47 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:47 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:58:47.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:47 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:58:47.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:47 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:58:47.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:47 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:58:48.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:48 smithi114 ceph-mon[30312]: pgmap v1251: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:48 smithi114 ceph-mon[42466]: pgmap v1251: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:48 smithi119 ceph-mon[33714]: pgmap v1251: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:49.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:58:50.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:50 smithi114 ceph-mon[30312]: pgmap v1252: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:50 smithi114 ceph-mon[42466]: pgmap v1252: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:50 smithi119 ceph-mon[33714]: pgmap v1252: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:52.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:52 smithi114 ceph-mon[30312]: pgmap v1253: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:52 smithi114 ceph-mon[42466]: pgmap v1253: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:52.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:52 smithi119 ceph-mon[33714]: pgmap v1253: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:58:52.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:58:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:58:53.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:53.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:58:54.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:54 smithi114 ceph-mon[30312]: pgmap v1254: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:54.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:58:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:58:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:58:54.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:54 smithi114 ceph-mon[42466]: pgmap v1254: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:54 smithi119 ceph-mon[33714]: pgmap v1254: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:56 smithi114 ceph-mon[30312]: pgmap v1255: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:56.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:56 smithi114 ceph-mon[42466]: pgmap v1255: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:56 smithi119 ceph-mon[33714]: pgmap v1255: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:58:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:58:58 smithi114 ceph-mon[30312]: pgmap v1256: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:58.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:58:58 smithi114 ceph-mon[42466]: pgmap v1256: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:58:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:58:58 smithi119 ceph-mon[33714]: pgmap v1256: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:00.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:00 smithi114 ceph-mon[30312]: pgmap v1257: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:00.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:00 smithi114 ceph-mon[42466]: pgmap v1257: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:00 smithi119 ceph-mon[33714]: pgmap v1257: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:02.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:02 smithi114 ceph-mon[30312]: pgmap v1258: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:02 smithi114 ceph-mon[42466]: pgmap v1258: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:02 smithi119 ceph-mon[33714]: pgmap v1258: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:02.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:59:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:59:03.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:03.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:03.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:04.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:04 smithi114 ceph-mon[30312]: pgmap v1259: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:04.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:04.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:59:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:59:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:04.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:04 smithi114 ceph-mon[42466]: pgmap v1259: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:04.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:04 smithi119 ceph-mon[33714]: pgmap v1259: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:06 smithi114 ceph-mon[30312]: pgmap v1260: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:06 smithi114 ceph-mon[42466]: pgmap v1260: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:06 smithi119 ceph-mon[33714]: pgmap v1260: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:08 smithi114 ceph-mon[30312]: pgmap v1261: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:08 smithi114 ceph-mon[42466]: pgmap v1261: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:08 smithi119 ceph-mon[33714]: pgmap v1261: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:10 smithi114 ceph-mon[30312]: pgmap v1262: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:10 smithi114 ceph-mon[42466]: pgmap v1262: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:10 smithi119 ceph-mon[33714]: pgmap v1262: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:12 smithi114 ceph-mon[30312]: pgmap v1263: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:12.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:12 smithi114 ceph-mon[42466]: pgmap v1263: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:12.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:12 smithi119 ceph-mon[33714]: pgmap v1263: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:12.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:59:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:59:13.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:13.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:13.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:14.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:14 smithi114 ceph-mon[30312]: pgmap v1264: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:14.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:14 smithi114 ceph-mon[42466]: pgmap v1264: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:14.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:59:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:59:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:14 smithi119 ceph-mon[33714]: pgmap v1264: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:16.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:16 smithi114 ceph-mon[30312]: pgmap v1265: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:16 smithi114 ceph-mon[42466]: pgmap v1265: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:16.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:16 smithi119 ceph-mon[33714]: pgmap v1265: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:18.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:18 smithi114 ceph-mon[30312]: pgmap v1266: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:18.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:18 smithi114 ceph-mon[42466]: pgmap v1266: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:18 smithi119 ceph-mon[33714]: pgmap v1266: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:19.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:19.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:19.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:20 smithi114 ceph-mon[30312]: pgmap v1267: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:20 smithi114 ceph-mon[42466]: pgmap v1267: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:20 smithi119 ceph-mon[33714]: pgmap v1267: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:22.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:22 smithi114 ceph-mon[30312]: pgmap v1268: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:22 smithi114 ceph-mon[42466]: pgmap v1268: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:22.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:22 smithi119 ceph-mon[33714]: pgmap v1268: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:22.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:59:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:59:23.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:24 smithi119 ceph-mon[33714]: pgmap v1269: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:24 smithi114 ceph-mon[30312]: pgmap v1269: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:24 smithi114 ceph-mon[42466]: pgmap v1269: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:59:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:59:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:26.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:26 smithi119 ceph-mon[33714]: pgmap v1270: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:26.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:26 smithi114 ceph-mon[30312]: pgmap v1270: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:26.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:26 smithi114 ceph-mon[42466]: pgmap v1270: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:28 smithi119 ceph-mon[33714]: pgmap v1271: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:28.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:28 smithi114 ceph-mon[30312]: pgmap v1271: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:28 smithi114 ceph-mon[42466]: pgmap v1271: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:30 smithi119 ceph-mon[33714]: pgmap v1272: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:30.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:30 smithi114 ceph-mon[30312]: pgmap v1272: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:30 smithi114 ceph-mon[42466]: pgmap v1272: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:32.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:32 smithi114 ceph-mon[30312]: pgmap v1273: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:32.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:32 smithi114 ceph-mon[42466]: pgmap v1273: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:32 smithi119 ceph-mon[33714]: pgmap v1273: 292 pgs: 292 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:32.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:59:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:59:33.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:33.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:33.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:34 smithi119 ceph-mon[33714]: pgmap v1274: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:34 smithi114 ceph-mon[30312]: pgmap v1274: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:34 smithi114 ceph-mon[42466]: pgmap v1274: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:59:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:59:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:36.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:36 smithi119 ceph-mon[33714]: pgmap v1275: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:36.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:36 smithi114 ceph-mon[30312]: pgmap v1275: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:36 smithi114 ceph-mon[42466]: pgmap v1275: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:38.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:38 smithi119 ceph-mon[33714]: pgmap v1276: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:38.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:38 smithi114 ceph-mon[30312]: pgmap v1276: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:38 smithi114 ceph-mon[42466]: pgmap v1276: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:40 smithi119 ceph-mon[33714]: pgmap v1277: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:40 smithi114 ceph-mon[30312]: pgmap v1277: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:40 smithi114 ceph-mon[42466]: pgmap v1277: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:42 smithi119 ceph-mon[33714]: pgmap v1278: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:42.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:59:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:59:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:42 smithi114 ceph-mon[30312]: pgmap v1278: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:42 smithi114 ceph-mon[42466]: pgmap v1278: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:43.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:43.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:44 smithi119 ceph-mon[33714]: pgmap v1279: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:44 smithi114 ceph-mon[30312]: pgmap v1279: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:44 smithi114 ceph-mon[42466]: pgmap v1279: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T11:59:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:59:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:59:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:46 smithi119 ceph-mon[33714]: pgmap v1280: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:46 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:46 smithi114 ceph-mon[30312]: pgmap v1280: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:46.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:46 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:46 smithi114 ceph-mon[42466]: pgmap v1280: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:46 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:59:47.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:47 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:47.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:47 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:47.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:47 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:59:47.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:47 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:47.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:47 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:47.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:47 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:59:47.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:47 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:59:47.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:47 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:59:47.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:47 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T11:59:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:48 smithi119 ceph-mon[33714]: pgmap v1281: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:48.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:48 smithi114 ceph-mon[30312]: pgmap v1281: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:48 smithi114 ceph-mon[42466]: pgmap v1281: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:49.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:49.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:59:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:50 smithi119 ceph-mon[33714]: pgmap v1282: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:50 smithi114 ceph-mon[30312]: pgmap v1282: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:50 smithi114 ceph-mon[42466]: pgmap v1282: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:52.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:52 smithi119 ceph-mon[33714]: pgmap v1283: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:52.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 11:59:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T11:59:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:52 smithi114 ceph-mon[30312]: pgmap v1283: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:52 smithi114 ceph-mon[42466]: pgmap v1283: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:53.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:59:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:54 smithi114 ceph-mon[30312]: pgmap v1284: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:54 smithi114 ceph-mon[42466]: pgmap v1284: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 11:59:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:11:59:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:59:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:54 smithi119 ceph-mon[33714]: pgmap v1284: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:56.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:56 smithi114 ceph-mon[30312]: pgmap v1285: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:56 smithi114 ceph-mon[42466]: pgmap v1285: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:57.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:56 smithi119 ceph-mon[33714]: pgmap v1285: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:59:58.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 11:59:58 smithi114 ceph-mon[30312]: pgmap v1286: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 11:59:58 smithi114 ceph-mon[42466]: pgmap v1286: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:59:59.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 11:59:58 smithi119 ceph-mon[33714]: pgmap v1286: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:00 smithi114 ceph-mon[42466]: pgmap v1287: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:00 smithi114 ceph-mon[42466]: overall HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T12:00:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:00 smithi114 ceph-mon[30312]: pgmap v1287: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:00 smithi114 ceph-mon[30312]: overall HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T12:00:01.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:00 smithi119 ceph-mon[33714]: pgmap v1287: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:01.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:00 smithi119 ceph-mon[33714]: overall HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T12:00:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:02 smithi119 ceph-mon[33714]: pgmap v1288: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:02.806 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:00:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:00:02.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:02 smithi114 ceph-mon[42466]: pgmap v1288: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:02.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:02 smithi114 ceph-mon[30312]: pgmap v1288: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:03.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:03.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:04.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:04 smithi114 ceph-mon[30312]: pgmap v1289: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:04 smithi114 ceph-mon[42466]: pgmap v1289: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:04.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:00:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:00:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:04 smithi119 ceph-mon[33714]: pgmap v1289: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:06 smithi114 ceph-mon[30312]: pgmap v1290: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:06 smithi114 ceph-mon[42466]: pgmap v1290: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:06 smithi119 ceph-mon[33714]: pgmap v1290: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:08 smithi114 ceph-mon[30312]: pgmap v1291: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:08 smithi114 ceph-mon[42466]: pgmap v1291: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:08 smithi119 ceph-mon[33714]: pgmap v1291: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:10.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:10 smithi114 ceph-mon[30312]: pgmap v1292: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:10 smithi114 ceph-mon[42466]: pgmap v1292: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:11.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:10 smithi119 ceph-mon[33714]: pgmap v1292: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:12.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:12 smithi119 ceph-mon[33714]: pgmap v1293: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:00:12.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:00:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:00:12.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:12 smithi114 ceph-mon[30312]: pgmap v1293: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:00:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:12 smithi114 ceph-mon[42466]: pgmap v1293: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:00:13.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:13.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:14.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:14 smithi114 ceph-mon[30312]: pgmap v1294: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:14 smithi114 ceph-mon[42466]: pgmap v1294: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:00:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:00:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:14 smithi119 ceph-mon[33714]: pgmap v1294: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:16 smithi114 ceph-mon[30312]: pgmap v1295: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:16 smithi114 ceph-mon[42466]: pgmap v1295: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:16 smithi119 ceph-mon[33714]: pgmap v1295: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:18 smithi119 ceph-mon[33714]: pgmap v1296: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:18 smithi114 ceph-mon[42466]: pgmap v1296: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:18 smithi114 ceph-mon[30312]: pgmap v1296: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:19.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:20.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:20 smithi114 ceph-mon[30312]: pgmap v1297: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:20 smithi114 ceph-mon[42466]: pgmap v1297: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:20 smithi119 ceph-mon[33714]: pgmap v1297: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:22.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:22 smithi119 ceph-mon[33714]: pgmap v1298: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:00:22.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:00:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:00:22.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:22 smithi114 ceph-mon[30312]: pgmap v1298: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:00:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:22 smithi114 ceph-mon[42466]: pgmap v1298: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:00:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:24 smithi114 ceph-mon[30312]: pgmap v1299: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:24 smithi114 ceph-mon[42466]: pgmap v1299: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:00:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:00:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:24 smithi119 ceph-mon[33714]: pgmap v1299: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:26.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:26 smithi114 ceph-mon[30312]: pgmap v1300: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:26 smithi114 ceph-mon[42466]: pgmap v1300: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:27.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:26 smithi119 ceph-mon[33714]: pgmap v1300: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:28.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:28 smithi114 ceph-mon[30312]: pgmap v1301: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:28 smithi114 ceph-mon[42466]: pgmap v1301: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:28 smithi119 ceph-mon[33714]: pgmap v1301: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:30 smithi114 ceph-mon[42466]: pgmap v1302: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:30.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:30 smithi114 ceph-mon[30312]: pgmap v1302: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:30 smithi119 ceph-mon[33714]: pgmap v1302: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:32 smithi119 ceph-mon[33714]: pgmap v1303: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:32.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:00:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:00:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:32 smithi114 ceph-mon[30312]: pgmap v1303: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:32.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:32 smithi114 ceph-mon[42466]: pgmap v1303: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:33.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:33.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:34.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:34 smithi114 ceph-mon[42466]: pgmap v1304: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:34 smithi114 ceph-mon[30312]: pgmap v1304: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:00:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:00:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:34 smithi119 ceph-mon[33714]: pgmap v1304: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:36.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:36 smithi114 ceph-mon[30312]: pgmap v1305: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:36 smithi114 ceph-mon[42466]: pgmap v1305: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:36 smithi119 ceph-mon[33714]: pgmap v1305: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:38.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:38 smithi114 ceph-mon[30312]: pgmap v1306: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:38.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:38 smithi114 ceph-mon[42466]: pgmap v1306: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:39.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:38 smithi119 ceph-mon[33714]: pgmap v1306: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:40 smithi114 ceph-mon[30312]: pgmap v1307: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:40 smithi114 ceph-mon[42466]: pgmap v1307: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:41.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:40 smithi119 ceph-mon[33714]: pgmap v1307: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:41.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:41 smithi114 ceph-mon[30312]: pgmap v1308: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:41.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:41 smithi114 ceph-mon[42466]: pgmap v1308: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:42.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:41 smithi119 ceph-mon[33714]: pgmap v1308: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:42 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:42.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:00:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:00:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:42 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:42 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:43.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:43 smithi114 ceph-mon[30312]: pgmap v1309: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:43.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:43 smithi114 ceph-mon[42466]: pgmap v1309: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:43 smithi119 ceph-mon[33714]: pgmap v1309: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:00:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:00:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:46.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:46 smithi114 ceph-mon[30312]: pgmap v1310: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:46.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:46 smithi114 ceph-mon[42466]: pgmap v1310: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:46 smithi119 ceph-mon[33714]: pgmap v1310: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:00:46.553 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:00:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=cleanup t=2024-03-24T12:00:46.331892205Z level=info msg="Completed cleanup jobs" duration=6.418638ms 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[30312]: pgmap v1311: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[42466]: pgmap v1311: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:48.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:48.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:00:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:48 smithi119 ceph-mon[33714]: pgmap v1311: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:00:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:00:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:00:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:00:50.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:50 smithi114 ceph-mon[30312]: pgmap v1312: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:50.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:50.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:50 smithi114 ceph-mon[42466]: pgmap v1312: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:50.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:50.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:50 smithi119 ceph-mon[33714]: pgmap v1312: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:50.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:00:52.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:52 smithi114 ceph-mon[30312]: pgmap v1313: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:52.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:52 smithi114 ceph-mon[42466]: pgmap v1313: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:52.508 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:52 smithi119 ceph-mon[33714]: pgmap v1313: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:52.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:00:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:00:53.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:53.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:53.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:00:54.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:54 smithi114 ceph-mon[30312]: pgmap v1314: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:54.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:54 smithi114 ceph-mon[42466]: pgmap v1314: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:54.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:54 smithi119 ceph-mon[33714]: pgmap v1314: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:00:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:00:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:00:56.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:56 smithi114 ceph-mon[30312]: pgmap v1315: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:56.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:56 smithi114 ceph-mon[42466]: pgmap v1315: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:56.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:56 smithi119 ceph-mon[33714]: pgmap v1315: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:00:58.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:00:58 smithi114 ceph-mon[30312]: pgmap v1316: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:58.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:00:58 smithi114 ceph-mon[42466]: pgmap v1316: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:00:58.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:00:58 smithi119 ceph-mon[33714]: pgmap v1316: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:00.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:00 smithi114 ceph-mon[30312]: pgmap v1317: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:00.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:00 smithi114 ceph-mon[42466]: pgmap v1317: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:00.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:00 smithi119 ceph-mon[33714]: pgmap v1317: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:02.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:02 smithi114 ceph-mon[30312]: pgmap v1318: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:02.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:02 smithi114 ceph-mon[42466]: pgmap v1318: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:02.513 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:02 smithi119 ceph-mon[33714]: pgmap v1318: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:02.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:01:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:01:03.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:03.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:03.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:04 smithi114 ceph-mon[30312]: pgmap v1319: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:04.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:04 smithi114 ceph-mon[42466]: pgmap v1319: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:04 smithi119 ceph-mon[33714]: pgmap v1319: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:04.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:01:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:01:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:05.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:05.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:05.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:06.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:06 smithi114 ceph-mon[30312]: pgmap v1320: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:06.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:06 smithi114 ceph-mon[42466]: pgmap v1320: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:06.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:06 smithi119 ceph-mon[33714]: pgmap v1320: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:08.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:08 smithi119 ceph-mon[33714]: pgmap v1321: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:08 smithi114 ceph-mon[30312]: pgmap v1321: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:08 smithi114 ceph-mon[42466]: pgmap v1321: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:10 smithi119 ceph-mon[33714]: pgmap v1322: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:10 smithi114 ceph-mon[30312]: pgmap v1322: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:10 smithi114 ceph-mon[42466]: pgmap v1322: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:12.520 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:12 smithi119 ceph-mon[33714]: pgmap v1323: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:12.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:12 smithi114 ceph-mon[30312]: pgmap v1323: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:12.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:12 smithi114 ceph-mon[42466]: pgmap v1323: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:12.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:01:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:01:13.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:14 smithi119 ceph-mon[33714]: pgmap v1324: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:14.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:14 smithi114 ceph-mon[30312]: pgmap v1324: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:14.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:14 smithi114 ceph-mon[42466]: pgmap v1324: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:01:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:01:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:16 smithi119 ceph-mon[33714]: pgmap v1325: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:16.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:16 smithi114 ceph-mon[30312]: pgmap v1325: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:16 smithi114 ceph-mon[42466]: pgmap v1325: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:18 smithi119 ceph-mon[33714]: pgmap v1326: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:18.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:18 smithi114 ceph-mon[30312]: pgmap v1326: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:18.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:18 smithi114 ceph-mon[42466]: pgmap v1326: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:19.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:19.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:19.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:20.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:20 smithi119 ceph-mon[33714]: pgmap v1327: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:20.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:20 smithi114 ceph-mon[30312]: pgmap v1327: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:20 smithi114 ceph-mon[42466]: pgmap v1327: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:22.527 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:22 smithi119 ceph-mon[33714]: pgmap v1328: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:22.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:22 smithi114 ceph-mon[30312]: pgmap v1328: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:22 smithi114 ceph-mon[42466]: pgmap v1328: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:22.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:01:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:01:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:23.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:23.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:24 smithi119 ceph-mon[33714]: pgmap v1329: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:24.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:24 smithi114 ceph-mon[30312]: pgmap v1329: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:24.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:24 smithi114 ceph-mon[42466]: pgmap v1329: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:24.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:01:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:01:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:26 smithi119 ceph-mon[33714]: pgmap v1330: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:26.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:26 smithi114 ceph-mon[30312]: pgmap v1330: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:26.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:26 smithi114 ceph-mon[42466]: pgmap v1330: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:28 smithi119 ceph-mon[33714]: pgmap v1331: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:28.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:28 smithi114 ceph-mon[30312]: pgmap v1331: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:28.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:28 smithi114 ceph-mon[42466]: pgmap v1331: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:30 smithi119 ceph-mon[33714]: pgmap v1332: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:30.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:30 smithi114 ceph-mon[30312]: pgmap v1332: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:30.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:30 smithi114 ceph-mon[42466]: pgmap v1332: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:32.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:32 smithi119 ceph-mon[33714]: pgmap v1333: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:32.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:01:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:01:32.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:32 smithi114 ceph-mon[30312]: pgmap v1333: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:32.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:32 smithi114 ceph-mon[42466]: pgmap v1333: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:33.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:33.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:33.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:34.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:34 smithi114 ceph-mon[42466]: pgmap v1334: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:34.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:34 smithi114 ceph-mon[30312]: pgmap v1334: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:34 smithi119 ceph-mon[33714]: pgmap v1334: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:01:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:01:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:36 smithi114 ceph-mon[30312]: pgmap v1335: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:36 smithi114 ceph-mon[42466]: pgmap v1335: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:36 smithi119 ceph-mon[33714]: pgmap v1335: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:38.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:38 smithi114 ceph-mon[30312]: pgmap v1336: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:38.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:38 smithi114 ceph-mon[42466]: pgmap v1336: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:38.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:38 smithi119 ceph-mon[33714]: pgmap v1336: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:40.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:40 smithi114 ceph-mon[30312]: pgmap v1337: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:40.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:40 smithi114 ceph-mon[42466]: pgmap v1337: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:40 smithi119 ceph-mon[33714]: pgmap v1337: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:42.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:42 smithi114 ceph-mon[30312]: pgmap v1338: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:42.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:42 smithi114 ceph-mon[42466]: pgmap v1338: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:42 smithi119 ceph-mon[33714]: pgmap v1338: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:42.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:01:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:01:43.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:43.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:43.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:44.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:44 smithi114 ceph-mon[30312]: pgmap v1339: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:44.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:44 smithi114 ceph-mon[42466]: pgmap v1339: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:44 smithi119 ceph-mon[33714]: pgmap v1339: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:01:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:01:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:46 smithi114 ceph-mon[30312]: pgmap v1340: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:46 smithi114 ceph-mon[42466]: pgmap v1340: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:46 smithi119 ceph-mon[33714]: pgmap v1340: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:48.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:48 smithi114 ceph-mon[30312]: pgmap v1341: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:48.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:48 smithi114 ceph-mon[42466]: pgmap v1341: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:48 smithi119 ceph-mon[33714]: pgmap v1341: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:01:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:01:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:49.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:49.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:01:49.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:01:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:01:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:01:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:01:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:50 smithi119 ceph-mon[33714]: pgmap v1342: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:50 smithi114 ceph-mon[30312]: pgmap v1342: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:50 smithi114 ceph-mon[42466]: pgmap v1342: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:52.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:52 smithi119 ceph-mon[33714]: pgmap v1343: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:52.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:01:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:01:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:52 smithi114 ceph-mon[30312]: pgmap v1343: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:52 smithi114 ceph-mon[42466]: pgmap v1343: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:53.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:53.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:01:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:54 smithi119 ceph-mon[33714]: pgmap v1344: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:54 smithi114 ceph-mon[30312]: pgmap v1344: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:54 smithi114 ceph-mon[42466]: pgmap v1344: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:01:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:01:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:01:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:56 smithi119 ceph-mon[33714]: pgmap v1345: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:56 smithi114 ceph-mon[30312]: pgmap v1345: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:56 smithi114 ceph-mon[42466]: pgmap v1345: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:01:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:01:58 smithi119 ceph-mon[33714]: pgmap v1346: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:58.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:01:58 smithi114 ceph-mon[30312]: pgmap v1346: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:01:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:01:58 smithi114 ceph-mon[42466]: pgmap v1346: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:00 smithi119 ceph-mon[33714]: pgmap v1347: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:00 smithi114 ceph-mon[30312]: pgmap v1347: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:00 smithi114 ceph-mon[42466]: pgmap v1347: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:02.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:02 smithi114 ceph-mon[30312]: pgmap v1348: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:02.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:02 smithi114 ceph-mon[42466]: pgmap v1348: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:02 smithi119 ceph-mon[33714]: pgmap v1348: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:02.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:02:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:02:03.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:03.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:03.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:04 smithi119 ceph-mon[33714]: pgmap v1349: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:04 smithi114 ceph-mon[30312]: pgmap v1349: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:04 smithi114 ceph-mon[42466]: pgmap v1349: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:02:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:02:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:06 smithi119 ceph-mon[33714]: pgmap v1350: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:06 smithi114 ceph-mon[42466]: pgmap v1350: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:06 smithi114 ceph-mon[30312]: pgmap v1350: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:08 smithi119 ceph-mon[33714]: pgmap v1351: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:08 smithi114 ceph-mon[30312]: pgmap v1351: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:08 smithi114 ceph-mon[42466]: pgmap v1351: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:10.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:10 smithi114 ceph-mon[30312]: pgmap v1352: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:10 smithi114 ceph-mon[42466]: pgmap v1352: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:11.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:10 smithi119 ceph-mon[33714]: pgmap v1352: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:12.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:12 smithi114 ceph-mon[30312]: pgmap v1353: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:12 smithi114 ceph-mon[42466]: pgmap v1353: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:13.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:12 smithi119 ceph-mon[33714]: pgmap v1353: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:13.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:02:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:02:13.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:13.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:14.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:14 smithi114 ceph-mon[30312]: pgmap v1354: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:14 smithi114 ceph-mon[42466]: pgmap v1354: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:02:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:02:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:14 smithi119 ceph-mon[33714]: pgmap v1354: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:16 smithi114 ceph-mon[30312]: pgmap v1355: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:16 smithi114 ceph-mon[42466]: pgmap v1355: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:16 smithi119 ceph-mon[33714]: pgmap v1355: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:18.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:18 smithi114 ceph-mon[30312]: pgmap v1356: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:18 smithi114 ceph-mon[42466]: pgmap v1356: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:19.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:18 smithi119 ceph-mon[33714]: pgmap v1356: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:19.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:20.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:20 smithi114 ceph-mon[30312]: pgmap v1357: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:20 smithi114 ceph-mon[42466]: pgmap v1357: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:20 smithi119 ceph-mon[33714]: pgmap v1357: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:22 smithi114 ceph-mon[30312]: pgmap v1358: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:22 smithi114 ceph-mon[42466]: pgmap v1358: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:22 smithi119 ceph-mon[33714]: pgmap v1358: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:23.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:02:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:02:23.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:24 smithi114 ceph-mon[30312]: pgmap v1359: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:24 smithi114 ceph-mon[42466]: pgmap v1359: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:02:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:02:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:24 smithi119 ceph-mon[33714]: pgmap v1359: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:26 smithi114 ceph-mon[42466]: pgmap v1360: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:26.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:26 smithi114 ceph-mon[30312]: pgmap v1360: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:27.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:26 smithi119 ceph-mon[33714]: pgmap v1360: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:28.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:28 smithi114 ceph-mon[30312]: pgmap v1361: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:28 smithi114 ceph-mon[42466]: pgmap v1361: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:28 smithi119 ceph-mon[33714]: pgmap v1361: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:30.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:30 smithi114 ceph-mon[30312]: pgmap v1362: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:30 smithi114 ceph-mon[42466]: pgmap v1362: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:30 smithi119 ceph-mon[33714]: pgmap v1362: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:32.855 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:32 smithi119 ceph-mon[33714]: pgmap v1363: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:32.856 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:02:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:02:32.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:32 smithi114 ceph-mon[30312]: pgmap v1363: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:32.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:32 smithi114 ceph-mon[42466]: pgmap v1363: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:33.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:33.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:34.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:34 smithi114 ceph-mon[42466]: pgmap v1364: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:34 smithi114 ceph-mon[30312]: pgmap v1364: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:02:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:02:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:34 smithi119 ceph-mon[33714]: pgmap v1364: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:36.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:36 smithi114 ceph-mon[30312]: pgmap v1365: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:36 smithi114 ceph-mon[42466]: pgmap v1365: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:36 smithi119 ceph-mon[33714]: pgmap v1365: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:38 smithi114 ceph-mon[30312]: pgmap v1366: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:38 smithi114 ceph-mon[42466]: pgmap v1366: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:39.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:38 smithi119 ceph-mon[33714]: pgmap v1366: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:40 smithi114 ceph-mon[30312]: pgmap v1367: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:40 smithi114 ceph-mon[42466]: pgmap v1367: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:41.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:40 smithi119 ceph-mon[33714]: pgmap v1367: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:42 smithi114 ceph-mon[30312]: pgmap v1368: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:42 smithi114 ceph-mon[42466]: pgmap v1368: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:43.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:42 smithi119 ceph-mon[33714]: pgmap v1368: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:43.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:02:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:02:43.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:43.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:44.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:44.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:44 smithi114 ceph-mon[30312]: pgmap v1369: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:44 smithi114 ceph-mon[42466]: pgmap v1369: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:02:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:02:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:44 smithi119 ceph-mon[33714]: pgmap v1369: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:46 smithi114 ceph-mon[30312]: pgmap v1370: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:46 smithi114 ceph-mon[42466]: pgmap v1370: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:47.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:46 smithi119 ceph-mon[33714]: pgmap v1370: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:48.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:48 smithi114 ceph-mon[30312]: pgmap v1371: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:48.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:48 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:48 smithi114 ceph-mon[42466]: pgmap v1371: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:48 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:49.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:48 smithi119 ceph-mon[33714]: pgmap v1371: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:49.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:48 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:02:49.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:49.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:02:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:50 smithi114 ceph-mon[30312]: pgmap v1372: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:50 smithi114 ceph-mon[42466]: pgmap v1372: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:51.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:50 smithi119 ceph-mon[33714]: pgmap v1372: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:52 smithi114 ceph-mon[30312]: pgmap v1373: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:52 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:52 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:52 smithi114 ceph-mon[42466]: pgmap v1373: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:52 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:52 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:53.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:02:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:02:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:52 smithi119 ceph-mon[33714]: pgmap v1373: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:02:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:52 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:52 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:53 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:53.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:53 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:53.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:53.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:53 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:53.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:53 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:02:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:53 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:53 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[30312]: pgmap v1374: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[42466]: pgmap v1374: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:54.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:54.989 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:02:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:02:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:02:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:54 smithi119 ceph-mon[33714]: pgmap v1374: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:54 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:02:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:54 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:02:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:54 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:02:56.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:56 smithi114 ceph-mon[30312]: pgmap v1375: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:56 smithi114 ceph-mon[42466]: pgmap v1375: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:57.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:56 smithi119 ceph-mon[33714]: pgmap v1375: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:02:57.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:02:57 smithi114 ceph-mon[30312]: pgmap v1376: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:57.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:02:57 smithi114 ceph-mon[42466]: pgmap v1376: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:02:58.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:02:57 smithi119 ceph-mon[33714]: pgmap v1376: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:00.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:00 smithi114 ceph-mon[30312]: pgmap v1377: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:00.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:00 smithi114 ceph-mon[42466]: pgmap v1377: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:00.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:00 smithi119 ceph-mon[33714]: pgmap v1377: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:02.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:02 smithi119 ceph-mon[33714]: pgmap v1378: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:02.562 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:02 smithi114 ceph-mon[30312]: pgmap v1378: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:02.563 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:02 smithi114 ceph-mon[42466]: pgmap v1378: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:03.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:03:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:03:03.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:03.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:03.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:04.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:04 smithi119 ceph-mon[33714]: pgmap v1379: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:04 smithi114 ceph-mon[30312]: pgmap v1379: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:04 smithi114 ceph-mon[42466]: pgmap v1379: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:03:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:03:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:05.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:05.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:05.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:06.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:06 smithi119 ceph-mon[33714]: pgmap v1380: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:06 smithi114 ceph-mon[42466]: pgmap v1380: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:06 smithi114 ceph-mon[30312]: pgmap v1380: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:08.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:08 smithi119 ceph-mon[33714]: pgmap v1381: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:08.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:08 smithi114 ceph-mon[30312]: pgmap v1381: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:08 smithi114 ceph-mon[42466]: pgmap v1381: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:10 smithi119 ceph-mon[33714]: pgmap v1382: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:10 smithi114 ceph-mon[30312]: pgmap v1382: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:10 smithi114 ceph-mon[42466]: pgmap v1382: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:12.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:12 smithi119 ceph-mon[33714]: pgmap v1383: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:12.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:12 smithi114 ceph-mon[30312]: pgmap v1383: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:12.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:12 smithi114 ceph-mon[42466]: pgmap v1383: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:13.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:03:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:03:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:13 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:13.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:13 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:13.563 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:13 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:14 smithi119 ceph-mon[33714]: pgmap v1384: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:14.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:14 smithi114 ceph-mon[30312]: pgmap v1384: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:14.653 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:14 smithi114 ceph-mon[42466]: pgmap v1384: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:14.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:03:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:03:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:16 smithi119 ceph-mon[33714]: pgmap v1385: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:16.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:16 smithi114 ceph-mon[30312]: pgmap v1385: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:16 smithi114 ceph-mon[42466]: pgmap v1385: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:18.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:18 smithi119 ceph-mon[33714]: pgmap v1386: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:18.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:18 smithi114 ceph-mon[30312]: pgmap v1386: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:18.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:18 smithi114 ceph-mon[42466]: pgmap v1386: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:20.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:20 smithi119 ceph-mon[33714]: pgmap v1387: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:20.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:20 smithi114 ceph-mon[30312]: pgmap v1387: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:20 smithi114 ceph-mon[42466]: pgmap v1387: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:22.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:22 smithi119 ceph-mon[33714]: pgmap v1388: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:22.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:22 smithi114 ceph-mon[30312]: pgmap v1388: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:22 smithi114 ceph-mon[42466]: pgmap v1388: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:23.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:03:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:03:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:23 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:23.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:23 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:23.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:23 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:24 smithi119 ceph-mon[33714]: pgmap v1389: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:24.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:24 smithi114 ceph-mon[30312]: pgmap v1389: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:24.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:24 smithi114 ceph-mon[42466]: pgmap v1389: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:24.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:03:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:03:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:26.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:26 smithi119 ceph-mon[33714]: pgmap v1390: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:26.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:26 smithi114 ceph-mon[30312]: pgmap v1390: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:26.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:26 smithi114 ceph-mon[42466]: pgmap v1390: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:28.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:28 smithi119 ceph-mon[33714]: pgmap v1391: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:28.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:28 smithi114 ceph-mon[30312]: pgmap v1391: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:28.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:28 smithi114 ceph-mon[42466]: pgmap v1391: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:30.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:30 smithi119 ceph-mon[33714]: pgmap v1392: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:30.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:30 smithi114 ceph-mon[30312]: pgmap v1392: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:30.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:30 smithi114 ceph-mon[42466]: pgmap v1392: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:32.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:32 smithi119 ceph-mon[33714]: pgmap v1393: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:32.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:32 smithi114 ceph-mon[30312]: pgmap v1393: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:32.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:32 smithi114 ceph-mon[42466]: pgmap v1393: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:33.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:03:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:03:33.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:33.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:33.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:34.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:34 smithi119 ceph-mon[33714]: pgmap v1394: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:34.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:34 smithi114 ceph-mon[30312]: pgmap v1394: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:34.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:34 smithi114 ceph-mon[42466]: pgmap v1394: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:34.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:03:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:03:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:35.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:35.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:35.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:36 smithi114 ceph-mon[30312]: pgmap v1395: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:36 smithi114 ceph-mon[42466]: pgmap v1395: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:36 smithi119 ceph-mon[33714]: pgmap v1395: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:38.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:38 smithi114 ceph-mon[30312]: pgmap v1396: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:38.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:38 smithi114 ceph-mon[42466]: pgmap v1396: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:38.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:38 smithi119 ceph-mon[33714]: pgmap v1396: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:40.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:40 smithi114 ceph-mon[30312]: pgmap v1397: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:40.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:40 smithi114 ceph-mon[42466]: pgmap v1397: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:40 smithi119 ceph-mon[33714]: pgmap v1397: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:42.659 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:42 smithi119 ceph-mon[33714]: pgmap v1398: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:42.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:42 smithi114 ceph-mon[30312]: pgmap v1398: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:42.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:42 smithi114 ceph-mon[42466]: pgmap v1398: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:03:43.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:03:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:03:43.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:43.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:43.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:44.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:44 smithi114 ceph-mon[30312]: pgmap v1399: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:44.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:44 smithi114 ceph-mon[42466]: pgmap v1399: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:44 smithi119 ceph-mon[33714]: pgmap v1399: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:44.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:03:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:03:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:46 smithi114 ceph-mon[42466]: pgmap v1400: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:46 smithi114 ceph-mon[30312]: pgmap v1400: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:46 smithi119 ceph-mon[33714]: pgmap v1400: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:48.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:48 smithi114 ceph-mon[30312]: pgmap v1401: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:48 smithi114 ceph-mon[42466]: pgmap v1401: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:48 smithi119 ceph-mon[33714]: pgmap v1401: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:50 smithi114 ceph-mon[30312]: pgmap v1402: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:50 smithi114 ceph-mon[42466]: pgmap v1402: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:50.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:50 smithi119 ceph-mon[33714]: pgmap v1402: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:03:52.661 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:52 smithi119 ceph-mon[33714]: pgmap v1403: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:52.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:52 smithi114 ceph-mon[30312]: pgmap v1403: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:52 smithi114 ceph-mon[42466]: pgmap v1403: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:53.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:03:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:03:53.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:53.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:53.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:03:54.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:54 smithi114 ceph-mon[30312]: pgmap v1404: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:54.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:54 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:03:54.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:54 smithi114 ceph-mon[42466]: pgmap v1404: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:54.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:54 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:03:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:54 smithi119 ceph-mon[33714]: pgmap v1404: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:54 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:03:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:03:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:03:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:03:55.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:55 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:03:55.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:55 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:03:55.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:55 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:03:55.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:55 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:03:55.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:55 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:03:55.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:55 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:03:55.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:55 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:03:55.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:55 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:03:55.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:55 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:03:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:56 smithi119 ceph-mon[33714]: pgmap v1405: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:56.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:56 smithi114 ceph-mon[30312]: pgmap v1405: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:56 smithi114 ceph-mon[42466]: pgmap v1405: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:03:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:03:58 smithi119 ceph-mon[33714]: pgmap v1406: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:58.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:03:58 smithi114 ceph-mon[30312]: pgmap v1406: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:03:58.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:03:58 smithi114 ceph-mon[42466]: pgmap v1406: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:00 smithi119 ceph-mon[33714]: pgmap v1407: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:00 smithi114 ceph-mon[30312]: pgmap v1407: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:00 smithi114 ceph-mon[42466]: pgmap v1407: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:02.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:02 smithi114 ceph-mon[30312]: pgmap v1408: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:02.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:02 smithi114 ceph-mon[42466]: pgmap v1408: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:02 smithi119 ceph-mon[33714]: pgmap v1408: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:02.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:04:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:04:03.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:03 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:03.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:03 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:03.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:03 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:04 smithi119 ceph-mon[33714]: pgmap v1409: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:04 smithi114 ceph-mon[30312]: pgmap v1409: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:04:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:04:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:04 smithi114 ceph-mon[42466]: pgmap v1409: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:06 smithi119 ceph-mon[33714]: pgmap v1410: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:06 smithi114 ceph-mon[30312]: pgmap v1410: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:06 smithi114 ceph-mon[42466]: pgmap v1410: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:08 smithi119 ceph-mon[33714]: pgmap v1411: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:08 smithi114 ceph-mon[30312]: pgmap v1411: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:08 smithi114 ceph-mon[42466]: pgmap v1411: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:10 smithi119 ceph-mon[33714]: pgmap v1412: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:10.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:10 smithi114 ceph-mon[30312]: pgmap v1412: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:10 smithi114 ceph-mon[42466]: pgmap v1412: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:12.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:12 smithi119 ceph-mon[33714]: pgmap v1413: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:12.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:04:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:04:12.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:12 smithi114 ceph-mon[30312]: pgmap v1413: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:12 smithi114 ceph-mon[42466]: pgmap v1413: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:14 smithi119 ceph-mon[33714]: pgmap v1414: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:14.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:14.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:14 smithi114 ceph-mon[30312]: pgmap v1414: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:14 smithi114 ceph-mon[42466]: pgmap v1414: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:04:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:04:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:16.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:16 smithi119 ceph-mon[33714]: pgmap v1415: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:16 smithi114 ceph-mon[30312]: pgmap v1415: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:16 smithi114 ceph-mon[42466]: pgmap v1415: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:18 smithi119 ceph-mon[33714]: pgmap v1416: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:18.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:18 smithi114 ceph-mon[30312]: pgmap v1416: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:18 smithi114 ceph-mon[42466]: pgmap v1416: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:19.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:20.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:20 smithi114 ceph-mon[30312]: pgmap v1417: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:20 smithi114 ceph-mon[42466]: pgmap v1417: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:20 smithi119 ceph-mon[33714]: pgmap v1417: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:22.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:22 smithi114 ceph-mon[30312]: pgmap v1418: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:22 smithi114 ceph-mon[42466]: pgmap v1418: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:22 smithi119 ceph-mon[33714]: pgmap v1418: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:23.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:04:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:04:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:24 smithi114 ceph-mon[30312]: pgmap v1419: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:24 smithi114 ceph-mon[42466]: pgmap v1419: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:04:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:04:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:25.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:24 smithi119 ceph-mon[33714]: pgmap v1419: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:26.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:26 smithi114 ceph-mon[30312]: pgmap v1420: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:26 smithi114 ceph-mon[42466]: pgmap v1420: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:27.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:26 smithi119 ceph-mon[33714]: pgmap v1420: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:28 smithi114 ceph-mon[30312]: pgmap v1421: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:28 smithi114 ceph-mon[42466]: pgmap v1421: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:28 smithi119 ceph-mon[33714]: pgmap v1421: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:30 smithi114 ceph-mon[30312]: pgmap v1422: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:30 smithi114 ceph-mon[42466]: pgmap v1422: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:30 smithi119 ceph-mon[33714]: pgmap v1422: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:32.855 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:32 smithi119 ceph-mon[33714]: pgmap v1423: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:32.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:04:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:04:32.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:32 smithi114 ceph-mon[30312]: pgmap v1423: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:32.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:32 smithi114 ceph-mon[42466]: pgmap v1423: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:34 smithi114 ceph-mon[30312]: pgmap v1424: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:04:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:04:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:34 smithi114 ceph-mon[42466]: pgmap v1424: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:34.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:34 smithi119 ceph-mon[33714]: pgmap v1424: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:36.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:36 smithi114 ceph-mon[30312]: pgmap v1425: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:36 smithi114 ceph-mon[42466]: pgmap v1425: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:36 smithi119 ceph-mon[33714]: pgmap v1425: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:38 smithi114 ceph-mon[30312]: pgmap v1426: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:38 smithi114 ceph-mon[42466]: pgmap v1426: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:39.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:38 smithi119 ceph-mon[33714]: pgmap v1426: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:40.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:40 smithi114 ceph-mon[30312]: pgmap v1427: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:40 smithi114 ceph-mon[42466]: pgmap v1427: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:41.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:40 smithi119 ceph-mon[33714]: pgmap v1427: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:42 smithi114 ceph-mon[30312]: pgmap v1428: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:04:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:42 smithi114 ceph-mon[42466]: pgmap v1428: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:04:43.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:42 smithi119 ceph-mon[33714]: pgmap v1428: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:04:43.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:04:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:04:44.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:44 smithi114 ceph-mon[30312]: pgmap v1429: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:04:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:04:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:44 smithi114 ceph-mon[42466]: pgmap v1429: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:44 smithi119 ceph-mon[33714]: pgmap v1429: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:46 smithi114 ceph-mon[30312]: pgmap v1430: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:46 smithi114 ceph-mon[42466]: pgmap v1430: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:47.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:46 smithi119 ceph-mon[33714]: pgmap v1430: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:48.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:48 smithi114 ceph-mon[30312]: pgmap v1431: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:48 smithi114 ceph-mon[42466]: pgmap v1431: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:49.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:48 smithi119 ceph-mon[33714]: pgmap v1431: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:49.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:49.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:50.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:04:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:50 smithi114 ceph-mon[30312]: pgmap v1432: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:50 smithi114 ceph-mon[42466]: pgmap v1432: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:50 smithi119 ceph-mon[33714]: pgmap v1432: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:52.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:52 smithi114 ceph-mon[30312]: pgmap v1433: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:04:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:52 smithi114 ceph-mon[42466]: pgmap v1433: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:04:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:52 smithi119 ceph-mon[33714]: pgmap v1433: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:04:53.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:04:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:04:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:54 smithi114 ceph-mon[30312]: pgmap v1434: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:54 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:04:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:04:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:04:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:04:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:54 smithi114 ceph-mon[42466]: pgmap v1434: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:54 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:04:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:04:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:54 smithi119 ceph-mon[33714]: pgmap v1434: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:54 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:04:55.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:55 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:04:55.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:55 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:04:55.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:55 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:04:55.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:55 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:04:55.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:55 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:04:55.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:55 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:04:56.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:55 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:04:56.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:55 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:04:56.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:55 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:04:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:56 smithi114 ceph-mon[30312]: pgmap v1435: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:56 smithi114 ceph-mon[42466]: pgmap v1435: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:57.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:56 smithi119 ceph-mon[33714]: pgmap v1435: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:04:58.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:04:58 smithi114 ceph-mon[30312]: pgmap v1436: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:04:58 smithi114 ceph-mon[42466]: pgmap v1436: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:04:59.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:04:58 smithi119 ceph-mon[33714]: pgmap v1436: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:00.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:00 smithi114 ceph-mon[42466]: pgmap v1437: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:00 smithi114 ceph-mon[30312]: pgmap v1437: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:01.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:00 smithi119 ceph-mon[33714]: pgmap v1437: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:02.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:02 smithi114 ceph-mon[30312]: pgmap v1438: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:05:02.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:02 smithi114 ceph-mon[42466]: pgmap v1438: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:05:03.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:02 smithi119 ceph-mon[33714]: pgmap v1438: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:05:03.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:05:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:05:04.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:04 smithi114 ceph-mon[42466]: pgmap v1439: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:04 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:04 smithi114 ceph-mon[30312]: pgmap v1439: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:04 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:05:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:05:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:04 smithi119 ceph-mon[33714]: pgmap v1439: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:04 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:06.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:06 smithi114 ceph-mon[42466]: pgmap v1440: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:06 smithi114 ceph-mon[30312]: pgmap v1440: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:06 smithi119 ceph-mon[33714]: pgmap v1440: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:08 smithi114 ceph-mon[42466]: pgmap v1441: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:08 smithi114 ceph-mon[30312]: pgmap v1441: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:08 smithi119 ceph-mon[33714]: pgmap v1441: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:10 smithi114 ceph-mon[42466]: pgmap v1442: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:10.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:10 smithi114 ceph-mon[30312]: pgmap v1442: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:11.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:10 smithi119 ceph-mon[33714]: pgmap v1442: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:12.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:12 smithi114 ceph-mon[30312]: pgmap v1443: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:05:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:12 smithi114 ceph-mon[42466]: pgmap v1443: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:05:13.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:12 smithi119 ceph-mon[33714]: pgmap v1443: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:05:13.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:05:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:05:14.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:14 smithi114 ceph-mon[30312]: pgmap v1444: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:14 smithi114 ceph-mon[42466]: pgmap v1444: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:05:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:05:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:14 smithi119 ceph-mon[33714]: pgmap v1444: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:16 smithi114 ceph-mon[30312]: pgmap v1445: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:16 smithi114 ceph-mon[42466]: pgmap v1445: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:16 smithi119 ceph-mon[33714]: pgmap v1445: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:18 smithi114 ceph-mon[30312]: pgmap v1446: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:18 smithi114 ceph-mon[42466]: pgmap v1446: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:19.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:18 smithi119 ceph-mon[33714]: pgmap v1446: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:19.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:20.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:20 smithi114 ceph-mon[30312]: pgmap v1447: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:20 smithi114 ceph-mon[42466]: pgmap v1447: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:21.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:20 smithi119 ceph-mon[33714]: pgmap v1447: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:22 smithi114 ceph-mon[30312]: pgmap v1448: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:22 smithi114 ceph-mon[42466]: pgmap v1448: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:22 smithi119 ceph-mon[33714]: pgmap v1448: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:23.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:05:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:05:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:24 smithi114 ceph-mon[30312]: pgmap v1449: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:05:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:05:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:24 smithi114 ceph-mon[42466]: pgmap v1449: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:25.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:24 smithi119 ceph-mon[33714]: pgmap v1449: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:26.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:26 smithi114 ceph-mon[30312]: pgmap v1450: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:26 smithi114 ceph-mon[42466]: pgmap v1450: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:27.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:26 smithi119 ceph-mon[33714]: pgmap v1450: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:28.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:28 smithi114 ceph-mon[30312]: pgmap v1451: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:28 smithi114 ceph-mon[42466]: pgmap v1451: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:28 smithi119 ceph-mon[33714]: pgmap v1451: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:30 smithi114 ceph-mon[30312]: pgmap v1452: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:30 smithi114 ceph-mon[42466]: pgmap v1452: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:30 smithi119 ceph-mon[33714]: pgmap v1452: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:32.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:32 smithi114 ceph-mon[30312]: pgmap v1453: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:32.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:32 smithi114 ceph-mon[42466]: pgmap v1453: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:33.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:32 smithi119 ceph-mon[33714]: pgmap v1453: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:33.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:05:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:05:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:34 smithi114 ceph-mon[30312]: pgmap v1454: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:34 smithi114 ceph-mon[42466]: pgmap v1454: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:05:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:05:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:34 smithi119 ceph-mon[33714]: pgmap v1454: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:36.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:36 smithi114 ceph-mon[30312]: pgmap v1455: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:36 smithi114 ceph-mon[42466]: pgmap v1455: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:36 smithi119 ceph-mon[33714]: pgmap v1455: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:38 smithi114 ceph-mon[30312]: pgmap v1456: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:38 smithi114 ceph-mon[42466]: pgmap v1456: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:39.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:38 smithi119 ceph-mon[33714]: pgmap v1456: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:40 smithi114 ceph-mon[30312]: pgmap v1457: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:40 smithi114 ceph-mon[42466]: pgmap v1457: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:41.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:40 smithi119 ceph-mon[33714]: pgmap v1457: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:42 smithi114 ceph-mon[30312]: pgmap v1458: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:42 smithi114 ceph-mon[42466]: pgmap v1458: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:43.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:42 smithi119 ceph-mon[33714]: pgmap v1458: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:43.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:05:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:05:44.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:44 smithi114 ceph-mon[30312]: pgmap v1459: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:44 smithi114 ceph-mon[42466]: pgmap v1459: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:05:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:05:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:44 smithi119 ceph-mon[33714]: pgmap v1459: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:46.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:45 smithi119 ceph-mon[33714]: pgmap v1460: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:46.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:45 smithi114 ceph-mon[30312]: pgmap v1460: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:46.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:45 smithi114 ceph-mon[42466]: pgmap v1460: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:48.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:48 smithi114 ceph-mon[30312]: pgmap v1461: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:48 smithi114 ceph-mon[42466]: pgmap v1461: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:48.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:48 smithi119 ceph-mon[33714]: pgmap v1461: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:50.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:50 smithi114 ceph-mon[30312]: pgmap v1462: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:50.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:50 smithi114 ceph-mon[42466]: pgmap v1462: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:50 smithi119 ceph-mon[33714]: pgmap v1462: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:05:52.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:52 smithi114 ceph-mon[30312]: pgmap v1463: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:52 smithi114 ceph-mon[42466]: pgmap v1463: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:52.758 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:52 smithi119 ceph-mon[33714]: pgmap v1463: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:53.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:05:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:05:54.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:54.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:54 smithi114 ceph-mon[30312]: pgmap v1464: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:54.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:54.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:54 smithi114 ceph-mon[42466]: pgmap v1464: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:05:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:54 smithi119 ceph-mon[33714]: pgmap v1464: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:54.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:05:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:05:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:05:55.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:55 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:05:55.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:55 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:05:55.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:55 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:05:56.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[30312]: pgmap v1465: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[42466]: pgmap v1465: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:56 smithi119 ceph-mon[33714]: pgmap v1465: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:05:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:57.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:57 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:05:57.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:57 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:05:57.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:57 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:57.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:57 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:05:57.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:57 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:05:57.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:57 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:57.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:57 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:05:57.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:57 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:05:57.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:57 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:05:58.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:05:58 smithi114 ceph-mon[30312]: pgmap v1466: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:58.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:05:58 smithi114 ceph-mon[42466]: pgmap v1466: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:05:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:05:58 smithi119 ceph-mon[33714]: pgmap v1466: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:00.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:00 smithi114 ceph-mon[30312]: pgmap v1467: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:00.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:00 smithi114 ceph-mon[42466]: pgmap v1467: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:00 smithi119 ceph-mon[33714]: pgmap v1467: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:02.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:02 smithi114 ceph-mon[30312]: pgmap v1468: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:06:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:02 smithi114 ceph-mon[42466]: pgmap v1468: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:06:02.769 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:02 smithi119 ceph-mon[33714]: pgmap v1468: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:06:03.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:06:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:06:04.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:04.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:04 smithi114 ceph-mon[30312]: pgmap v1469: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:04.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:04.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:04 smithi114 ceph-mon[42466]: pgmap v1469: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:04.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:06:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:06:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:04 smithi119 ceph-mon[33714]: pgmap v1469: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:05.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:05.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:05.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:06 smithi114 ceph-mon[42466]: pgmap v1470: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:06.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:06 smithi114 ceph-mon[30312]: pgmap v1470: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:06.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:06 smithi119 ceph-mon[33714]: pgmap v1470: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:08.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:08 smithi114 ceph-mon[30312]: pgmap v1471: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:08 smithi114 ceph-mon[42466]: pgmap v1471: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:08.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:08 smithi119 ceph-mon[33714]: pgmap v1471: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:10.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:10 smithi114 ceph-mon[30312]: pgmap v1472: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:10 smithi114 ceph-mon[42466]: pgmap v1472: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:10 smithi119 ceph-mon[33714]: pgmap v1472: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:12.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:12 smithi114 ceph-mon[30312]: pgmap v1473: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:06:12.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:12 smithi114 ceph-mon[42466]: pgmap v1473: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:06:12.779 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:12 smithi119 ceph-mon[33714]: pgmap v1473: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:06:13.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:06:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:06:14.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:14.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:14 smithi114 ceph-mon[30312]: pgmap v1474: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:14.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:14.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:14 smithi114 ceph-mon[42466]: pgmap v1474: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:14.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:06:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:06:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:14 smithi119 ceph-mon[33714]: pgmap v1474: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:16.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:16 smithi114 ceph-mon[30312]: pgmap v1475: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:16 smithi114 ceph-mon[42466]: pgmap v1475: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:16.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:16 smithi119 ceph-mon[33714]: pgmap v1475: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:18.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:18 smithi114 ceph-mon[30312]: pgmap v1476: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:18.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:18 smithi114 ceph-mon[42466]: pgmap v1476: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:18 smithi119 ceph-mon[33714]: pgmap v1476: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:20.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:20 smithi114 ceph-mon[30312]: pgmap v1477: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:20 smithi114 ceph-mon[42466]: pgmap v1477: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:20 smithi119 ceph-mon[33714]: pgmap v1477: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:22.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:22 smithi114 ceph-mon[30312]: pgmap v1478: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:22 smithi114 ceph-mon[42466]: pgmap v1478: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:22.790 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:22 smithi119 ceph-mon[33714]: pgmap v1478: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:23.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:06:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:06:24.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:24.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:24 smithi114 ceph-mon[30312]: pgmap v1479: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:24.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:06:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:06:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:24.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:24.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:24 smithi114 ceph-mon[42466]: pgmap v1479: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:24.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:24 smithi119 ceph-mon[33714]: pgmap v1479: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:26.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:26 smithi119 ceph-mon[33714]: pgmap v1480: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:26.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:26 smithi114 ceph-mon[30312]: pgmap v1480: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:26.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:26 smithi114 ceph-mon[42466]: pgmap v1480: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:28 smithi119 ceph-mon[33714]: pgmap v1481: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:28 smithi114 ceph-mon[30312]: pgmap v1481: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:28.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:28 smithi114 ceph-mon[42466]: pgmap v1481: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:30 smithi119 ceph-mon[33714]: pgmap v1482: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:30 smithi114 ceph-mon[30312]: pgmap v1482: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:30 smithi114 ceph-mon[42466]: pgmap v1482: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:32.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:32 smithi114 ceph-mon[30312]: pgmap v1483: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:32.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:32 smithi114 ceph-mon[42466]: pgmap v1483: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:32.801 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:32 smithi119 ceph-mon[33714]: pgmap v1483: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:33.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:06:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:06:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:34 smithi119 ceph-mon[33714]: pgmap v1484: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:34 smithi114 ceph-mon[30312]: pgmap v1484: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:34 smithi114 ceph-mon[42466]: pgmap v1484: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:06:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:06:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:35.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:35.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:35.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:36 smithi119 ceph-mon[33714]: pgmap v1485: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:36.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:36 smithi114 ceph-mon[30312]: pgmap v1485: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:36 smithi114 ceph-mon[42466]: pgmap v1485: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:38.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:38 smithi119 ceph-mon[33714]: pgmap v1486: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:38 smithi114 ceph-mon[30312]: pgmap v1486: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:38 smithi114 ceph-mon[42466]: pgmap v1486: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:40 smithi119 ceph-mon[33714]: pgmap v1487: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:40 smithi114 ceph-mon[30312]: pgmap v1487: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:40 smithi114 ceph-mon[42466]: pgmap v1487: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:42 smithi119 ceph-mon[33714]: pgmap v1488: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:42 smithi114 ceph-mon[30312]: pgmap v1488: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:42 smithi114 ceph-mon[42466]: pgmap v1488: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:43.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:06:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:06:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:44 smithi119 ceph-mon[33714]: pgmap v1489: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:44.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:44 smithi114 ceph-mon[42466]: pgmap v1489: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:44 smithi114 ceph-mon[30312]: pgmap v1489: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:06:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:06:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:46.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:46 smithi119 ceph-mon[33714]: pgmap v1490: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:46 smithi114 ceph-mon[30312]: pgmap v1490: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:46 smithi114 ceph-mon[42466]: pgmap v1490: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:48 smithi119 ceph-mon[33714]: pgmap v1491: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:48.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:48 smithi114 ceph-mon[30312]: pgmap v1491: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:48 smithi114 ceph-mon[42466]: pgmap v1491: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:50 smithi119 ceph-mon[33714]: pgmap v1492: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:50 smithi114 ceph-mon[30312]: pgmap v1492: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:50 smithi114 ceph-mon[42466]: pgmap v1492: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:06:52.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:52 smithi119 ceph-mon[33714]: pgmap v1493: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:52.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:52 smithi114 ceph-mon[30312]: pgmap v1493: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:52 smithi114 ceph-mon[42466]: pgmap v1493: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:53.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:06:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:06:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:54 smithi119 ceph-mon[33714]: pgmap v1494: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:54.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:54 smithi114 ceph-mon[42466]: pgmap v1494: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:06:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:54 smithi114 ceph-mon[30312]: pgmap v1494: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:06:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:06:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:06:56.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:56 smithi119 ceph-mon[33714]: pgmap v1495: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:56 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:06:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:56 smithi114 ceph-mon[30312]: pgmap v1495: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:56.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:56 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:06:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:56 smithi114 ceph-mon[42466]: pgmap v1495: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:06:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:56 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:06:57.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:57 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:06:57.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:57 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:06:57.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:57 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:06:57.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:57 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:06:57.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:57 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:06:57.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:57 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:06:57.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:57 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:06:57.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:57 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:06:57.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:57 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:06:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:06:58 smithi119 ceph-mon[33714]: pgmap v1496: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:58.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:06:58 smithi114 ceph-mon[30312]: pgmap v1496: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:06:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:06:58 smithi114 ceph-mon[42466]: pgmap v1496: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:00 smithi119 ceph-mon[33714]: pgmap v1497: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:00.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:00 smithi114 ceph-mon[30312]: pgmap v1497: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:00 smithi114 ceph-mon[42466]: pgmap v1497: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:02 smithi119 ceph-mon[33714]: pgmap v1498: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:07:02.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:02 smithi114 ceph-mon[30312]: pgmap v1498: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:07:02.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:02 smithi114 ceph-mon[42466]: pgmap v1498: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:07:03.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:07:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:07:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:04 smithi119 ceph-mon[33714]: pgmap v1499: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:04.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:04 smithi114 ceph-mon[42466]: pgmap v1499: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:04 smithi114 ceph-mon[30312]: pgmap v1499: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:07:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:07:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:05.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:06.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:06.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:06 smithi114 ceph-mon[30312]: pgmap v1500: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:06 smithi114 ceph-mon[42466]: pgmap v1500: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:06 smithi119 ceph-mon[33714]: pgmap v1500: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:08 smithi114 ceph-mon[30312]: pgmap v1501: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:08 smithi114 ceph-mon[42466]: pgmap v1501: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:08 smithi119 ceph-mon[33714]: pgmap v1501: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:10.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:10 smithi114 ceph-mon[30312]: pgmap v1502: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:10 smithi114 ceph-mon[42466]: pgmap v1502: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:11.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:10 smithi119 ceph-mon[33714]: pgmap v1502: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:12.833 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:12 smithi119 ceph-mon[33714]: pgmap v1503: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:12.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:12 smithi114 ceph-mon[30312]: pgmap v1503: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:12 smithi114 ceph-mon[42466]: pgmap v1503: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:13.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:07:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:07:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:14 smithi114 ceph-mon[42466]: pgmap v1504: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:14 smithi114 ceph-mon[30312]: pgmap v1504: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:07:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:07:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:14 smithi119 ceph-mon[33714]: pgmap v1504: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:16.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:16 smithi114 ceph-mon[30312]: pgmap v1505: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:16 smithi114 ceph-mon[42466]: pgmap v1505: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:17.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:16 smithi119 ceph-mon[33714]: pgmap v1505: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:18.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:18 smithi114 ceph-mon[30312]: pgmap v1506: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:18 smithi114 ceph-mon[42466]: pgmap v1506: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:19.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:18 smithi119 ceph-mon[33714]: pgmap v1506: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:19.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:19 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:19.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:19 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:20.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:19 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:20 smithi114 ceph-mon[30312]: pgmap v1507: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:20 smithi114 ceph-mon[42466]: pgmap v1507: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:20 smithi119 ceph-mon[33714]: pgmap v1507: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:22.844 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:22 smithi119 ceph-mon[33714]: pgmap v1508: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:22 smithi114 ceph-mon[30312]: pgmap v1508: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:22 smithi114 ceph-mon[42466]: pgmap v1508: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:23.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:07:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:07:24.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:24 smithi114 ceph-mon[30312]: pgmap v1509: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:07:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:07:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:24 smithi114 ceph-mon[42466]: pgmap v1509: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:24 smithi119 ceph-mon[33714]: pgmap v1509: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:26.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:26 smithi114 ceph-mon[30312]: pgmap v1510: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:26 smithi114 ceph-mon[42466]: pgmap v1510: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:27.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:26 smithi119 ceph-mon[33714]: pgmap v1510: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:28 smithi114 ceph-mon[30312]: pgmap v1511: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:28 smithi114 ceph-mon[42466]: pgmap v1511: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:28 smithi119 ceph-mon[33714]: pgmap v1511: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:30 smithi114 ceph-mon[30312]: pgmap v1512: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:30 smithi114 ceph-mon[42466]: pgmap v1512: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:30 smithi119 ceph-mon[33714]: pgmap v1512: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:32.855 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:32 smithi119 ceph-mon[33714]: pgmap v1513: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:32.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:32 smithi114 ceph-mon[30312]: pgmap v1513: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:32.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:32 smithi114 ceph-mon[42466]: pgmap v1513: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:33.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:07:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:07:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:34 smithi114 ceph-mon[30312]: pgmap v1514: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:34 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:34 smithi114 ceph-mon[42466]: pgmap v1514: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:34 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:07:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:07:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:34 smithi119 ceph-mon[33714]: pgmap v1514: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:34 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:36.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:36 smithi114 ceph-mon[30312]: pgmap v1515: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:36 smithi114 ceph-mon[42466]: pgmap v1515: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:36 smithi119 ceph-mon[33714]: pgmap v1515: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:38.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:38 smithi114 ceph-mon[30312]: pgmap v1516: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:38 smithi114 ceph-mon[42466]: pgmap v1516: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:39.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:38 smithi119 ceph-mon[33714]: pgmap v1516: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:40 smithi114 ceph-mon[30312]: pgmap v1517: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:40 smithi114 ceph-mon[42466]: pgmap v1517: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:41.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:40 smithi119 ceph-mon[33714]: pgmap v1517: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:42 smithi114 ceph-mon[30312]: pgmap v1518: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:07:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:42 smithi114 ceph-mon[42466]: pgmap v1518: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:07:43.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:42 smithi119 ceph-mon[33714]: pgmap v1518: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:07:43.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:07:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:07:44.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:44 smithi114 ceph-mon[30312]: pgmap v1519: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:44 smithi114 ceph-mon[42466]: pgmap v1519: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:07:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:07:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:44 smithi119 ceph-mon[33714]: pgmap v1519: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:46 smithi114 ceph-mon[30312]: pgmap v1520: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:46.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:46 smithi114 ceph-mon[42466]: pgmap v1520: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:47.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:46 smithi119 ceph-mon[33714]: pgmap v1520: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:48.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:48 smithi114 ceph-mon[30312]: pgmap v1521: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:48 smithi114 ceph-mon[42466]: pgmap v1521: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:49.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:48 smithi119 ceph-mon[33714]: pgmap v1521: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:49.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:49 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:49.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:49 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:50.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:49 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:07:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:50 smithi114 ceph-mon[30312]: pgmap v1522: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:50 smithi114 ceph-mon[42466]: pgmap v1522: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:50 smithi119 ceph-mon[33714]: pgmap v1522: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:52.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:52 smithi114 ceph-mon[30312]: pgmap v1523: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:52 smithi114 ceph-mon[42466]: pgmap v1523: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:52 smithi119 ceph-mon[33714]: pgmap v1523: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:53.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:07:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:07:54.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:54 smithi114 ceph-mon[42466]: pgmap v1524: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:54 smithi114 ceph-mon[30312]: pgmap v1524: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:07:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:07:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:07:55.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:07:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:54 smithi119 ceph-mon[33714]: pgmap v1524: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:56 smithi114 ceph-mon[30312]: pgmap v1525: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:56 smithi114 ceph-mon[42466]: pgmap v1525: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:57.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:56 smithi119 ceph-mon[33714]: pgmap v1525: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:07:57.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:57 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:07:57.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:57 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:07:58.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:57 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:07:58.712 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:07:58 smithi114 ceph-mon[30312]: pgmap v1526: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:58.712 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:07:58 smithi114 ceph-mon[42466]: pgmap v1526: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:07:59.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:07:58 smithi119 ceph-mon[33714]: pgmap v1526: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:00 smithi114 ceph-mon[30312]: pgmap v1527: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:00 smithi114 ceph-mon[42466]: pgmap v1527: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:01.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:00 smithi119 ceph-mon[33714]: pgmap v1527: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:02.638 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[30312]: pgmap v1528: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[42466]: pgmap v1528: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:08:02.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:02 smithi119 ceph-mon[33714]: pgmap v1528: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:08:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:08:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:08:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:08:03.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:08:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:08:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:04 smithi114 ceph-mon[30312]: pgmap v1529: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:04 smithi114 ceph-mon[42466]: pgmap v1529: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:04.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:04 smithi119 ceph-mon[33714]: pgmap v1529: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:04.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:08:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:08:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:05.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:05.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:05.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:06.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:06 smithi114 ceph-mon[30312]: pgmap v1530: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:06 smithi114 ceph-mon[42466]: pgmap v1530: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:06.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:06 smithi119 ceph-mon[33714]: pgmap v1530: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:08.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:08 smithi114 ceph-mon[30312]: pgmap v1531: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:08 smithi114 ceph-mon[42466]: pgmap v1531: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:08.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:08 smithi119 ceph-mon[33714]: pgmap v1531: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:10.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:10 smithi114 ceph-mon[30312]: pgmap v1532: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:10 smithi114 ceph-mon[42466]: pgmap v1532: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:10.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:10 smithi119 ceph-mon[33714]: pgmap v1532: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:12.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:12 smithi114 ceph-mon[30312]: pgmap v1533: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:12.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:12 smithi114 ceph-mon[42466]: pgmap v1533: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:12.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:12 smithi119 ceph-mon[33714]: pgmap v1533: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:13.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:08:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:08:14.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:14.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:14 smithi114 ceph-mon[30312]: pgmap v1534: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:14.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:14.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:14 smithi114 ceph-mon[42466]: pgmap v1534: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:14 smithi119 ceph-mon[33714]: pgmap v1534: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:14.910 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:08:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:08:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:16.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:16 smithi114 ceph-mon[30312]: pgmap v1535: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:16 smithi114 ceph-mon[42466]: pgmap v1535: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:16.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:16 smithi119 ceph-mon[33714]: pgmap v1535: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:18.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:18 smithi114 ceph-mon[30312]: pgmap v1536: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:18.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:18 smithi114 ceph-mon[42466]: pgmap v1536: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:18.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:18 smithi119 ceph-mon[33714]: pgmap v1536: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:20.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:20 smithi114 ceph-mon[30312]: pgmap v1537: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:20 smithi114 ceph-mon[42466]: pgmap v1537: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:20 smithi119 ceph-mon[33714]: pgmap v1537: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:22.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:22 smithi114 ceph-mon[30312]: pgmap v1538: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:22 smithi114 ceph-mon[42466]: pgmap v1538: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:22.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:22 smithi119 ceph-mon[33714]: pgmap v1538: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:23.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:08:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:08:24.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:24.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:24 smithi114 ceph-mon[30312]: pgmap v1539: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:24.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:24.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:24 smithi114 ceph-mon[42466]: pgmap v1539: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:24 smithi119 ceph-mon[33714]: pgmap v1539: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:24.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:08:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:08:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:26.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:26 smithi114 ceph-mon[30312]: pgmap v1540: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:26.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:26 smithi114 ceph-mon[42466]: pgmap v1540: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:26.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:26 smithi119 ceph-mon[33714]: pgmap v1540: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:28.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:28 smithi114 ceph-mon[30312]: pgmap v1541: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:28.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:28 smithi114 ceph-mon[42466]: pgmap v1541: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:28 smithi119 ceph-mon[33714]: pgmap v1541: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:30.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:30 smithi114 ceph-mon[30312]: pgmap v1542: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:30.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:30 smithi114 ceph-mon[42466]: pgmap v1542: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:30 smithi119 ceph-mon[33714]: pgmap v1542: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:32.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:32 smithi114 ceph-mon[30312]: pgmap v1543: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:32.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:32 smithi114 ceph-mon[42466]: pgmap v1543: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:32 smithi119 ceph-mon[33714]: pgmap v1543: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:33.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:08:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:08:34.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:34.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:34 smithi114 ceph-mon[30312]: pgmap v1544: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:34.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:34.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:34 smithi114 ceph-mon[42466]: pgmap v1544: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:34.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:08:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:08:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:34.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:34 smithi119 ceph-mon[33714]: pgmap v1544: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:35.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:35.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:35.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:36 smithi114 ceph-mon[30312]: pgmap v1545: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:36 smithi114 ceph-mon[42466]: pgmap v1545: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:36 smithi119 ceph-mon[33714]: pgmap v1545: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:38.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:38 smithi114 ceph-mon[30312]: pgmap v1546: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:38.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:38 smithi114 ceph-mon[42466]: pgmap v1546: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:38.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:38 smithi119 ceph-mon[33714]: pgmap v1546: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:40.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:40 smithi114 ceph-mon[30312]: pgmap v1547: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:40.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:40 smithi114 ceph-mon[42466]: pgmap v1547: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:40.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:40 smithi119 ceph-mon[33714]: pgmap v1547: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:42.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:42 smithi114 ceph-mon[30312]: pgmap v1548: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:42.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:42 smithi114 ceph-mon[42466]: pgmap v1548: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:42.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:42 smithi119 ceph-mon[33714]: pgmap v1548: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:43.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:08:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:08:44.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:44.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:44 smithi114 ceph-mon[30312]: pgmap v1549: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:44.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:08:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:08:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:44.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:44.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:44 smithi114 ceph-mon[42466]: pgmap v1549: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:44.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:44.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:44 smithi119 ceph-mon[33714]: pgmap v1549: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:46.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:46 smithi114 ceph-mon[30312]: pgmap v1550: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:46 smithi114 ceph-mon[42466]: pgmap v1550: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:46 smithi119 ceph-mon[33714]: pgmap v1550: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:48.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:48 smithi114 ceph-mon[30312]: pgmap v1551: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:48 smithi114 ceph-mon[42466]: pgmap v1551: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:48 smithi119 ceph-mon[33714]: pgmap v1551: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:50 smithi114 ceph-mon[30312]: pgmap v1552: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:50 smithi114 ceph-mon[42466]: pgmap v1552: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:50 smithi119 ceph-mon[33714]: pgmap v1552: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:08:52.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:52 smithi114 ceph-mon[30312]: pgmap v1553: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:52.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:52 smithi114 ceph-mon[42466]: pgmap v1553: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:52.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:52 smithi119 ceph-mon[33714]: pgmap v1553: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:53.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:08:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:08:54.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:54.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:53 smithi119 ceph-mon[33714]: pgmap v1554: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:54.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:54.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:53 smithi114 ceph-mon[30312]: pgmap v1554: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:54.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:08:54.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:53 smithi114 ceph-mon[42466]: pgmap v1554: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:08:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:08:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:08:56.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:56 smithi114 ceph-mon[30312]: pgmap v1555: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:56.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:56 smithi114 ceph-mon[42466]: pgmap v1555: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:56.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:56 smithi119 ceph-mon[33714]: pgmap v1555: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:08:58.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:08:58 smithi114 ceph-mon[30312]: pgmap v1556: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:58.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:08:58 smithi114 ceph-mon[42466]: pgmap v1556: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:08:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:08:58 smithi119 ceph-mon[33714]: pgmap v1556: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:00.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:00 smithi114 ceph-mon[30312]: pgmap v1557: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:00.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:00 smithi114 ceph-mon[42466]: pgmap v1557: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:00 smithi119 ceph-mon[33714]: pgmap v1557: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:02.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[30312]: pgmap v1558: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:02.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[42466]: pgmap v1558: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:09:02.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:09:02.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:02 smithi119 ceph-mon[33714]: pgmap v1558: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:09:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:09:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:09:02.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:09:03.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:09:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:09:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:04.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:04 smithi114 ceph-mon[30312]: pgmap v1559: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:04.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:04 smithi114 ceph-mon[42466]: pgmap v1559: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:04 smithi119 ceph-mon[33714]: pgmap v1559: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:09:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:09:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:05.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:05.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:05.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:06.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:06 smithi114 ceph-mon[30312]: pgmap v1560: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:06.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:06 smithi114 ceph-mon[42466]: pgmap v1560: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:06.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:06 smithi119 ceph-mon[33714]: pgmap v1560: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:08.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:08 smithi114 ceph-mon[30312]: pgmap v1561: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:08.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:08 smithi114 ceph-mon[42466]: pgmap v1561: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:08 smithi119 ceph-mon[33714]: pgmap v1561: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:10.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:10 smithi114 ceph-mon[30312]: pgmap v1562: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:10 smithi114 ceph-mon[42466]: pgmap v1562: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:10.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:10 smithi119 ceph-mon[33714]: pgmap v1562: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:12.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:12 smithi114 ceph-mon[30312]: pgmap v1563: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:12.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:12 smithi114 ceph-mon[42466]: pgmap v1563: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:12.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:12 smithi119 ceph-mon[33714]: pgmap v1563: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:13.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:09:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:09:14.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:14.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:14 smithi114 ceph-mon[30312]: pgmap v1564: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:14.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:09:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:09:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:14.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:14.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:14 smithi114 ceph-mon[42466]: pgmap v1564: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:14 smithi119 ceph-mon[33714]: pgmap v1564: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:16.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:16 smithi114 ceph-mon[30312]: pgmap v1565: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:16 smithi114 ceph-mon[42466]: pgmap v1565: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:16.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:16 smithi119 ceph-mon[33714]: pgmap v1565: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:18.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:18 smithi114 ceph-mon[30312]: pgmap v1566: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:18.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:18 smithi114 ceph-mon[42466]: pgmap v1566: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:18 smithi119 ceph-mon[33714]: pgmap v1566: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:20.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:20 smithi114 ceph-mon[30312]: pgmap v1567: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:20.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:20 smithi114 ceph-mon[42466]: pgmap v1567: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:20 smithi119 ceph-mon[33714]: pgmap v1567: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:22.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:22 smithi114 ceph-mon[30312]: pgmap v1568: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:22 smithi114 ceph-mon[42466]: pgmap v1568: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:22.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:22 smithi119 ceph-mon[33714]: pgmap v1568: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:23.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:09:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:09:24.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:24.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:24 smithi114 ceph-mon[30312]: pgmap v1569: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:24.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:24.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:24 smithi114 ceph-mon[42466]: pgmap v1569: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:24.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:09:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:09:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:24.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:24 smithi119 ceph-mon[33714]: pgmap v1569: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:26.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:26 smithi114 ceph-mon[30312]: pgmap v1570: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:26.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:26 smithi114 ceph-mon[42466]: pgmap v1570: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:26.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:26 smithi119 ceph-mon[33714]: pgmap v1570: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:28.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:28 smithi114 ceph-mon[30312]: pgmap v1571: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:28.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:28 smithi114 ceph-mon[42466]: pgmap v1571: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:28.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:28 smithi119 ceph-mon[33714]: pgmap v1571: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:30.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:30 smithi114 ceph-mon[30312]: pgmap v1572: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:30.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:30 smithi114 ceph-mon[42466]: pgmap v1572: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:30 smithi119 ceph-mon[33714]: pgmap v1572: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:32.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:32 smithi114 ceph-mon[30312]: pgmap v1573: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:32.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:32 smithi114 ceph-mon[42466]: pgmap v1573: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:32.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:32 smithi119 ceph-mon[33714]: pgmap v1573: 292 pgs: 292 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:33.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:09:32 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:09:34.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:34.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:34 smithi114 ceph-mon[42466]: pgmap v1574: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:34.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:34.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:34 smithi114 ceph-mon[30312]: pgmap v1574: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:34.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:09:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:09:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:34 smithi119 ceph-mon[33714]: pgmap v1574: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:35.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:35.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:35.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:36 smithi114 ceph-mon[30312]: pgmap v1575: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:36 smithi114 ceph-mon[42466]: pgmap v1575: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:36 smithi119 ceph-mon[33714]: pgmap v1575: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:38.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:38 smithi114 ceph-mon[30312]: pgmap v1576: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:38.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:38 smithi114 ceph-mon[42466]: pgmap v1576: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:38.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:38 smithi119 ceph-mon[33714]: pgmap v1576: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:40.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:40 smithi114 ceph-mon[42466]: pgmap v1577: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:40.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:40 smithi114 ceph-mon[30312]: pgmap v1577: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:40.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:40 smithi119 ceph-mon[33714]: pgmap v1577: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:42.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:42 smithi114 ceph-mon[30312]: pgmap v1578: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:42.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:42 smithi114 ceph-mon[42466]: pgmap v1578: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:42 smithi119 ceph-mon[33714]: pgmap v1578: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:43.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:09:42 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:09:44.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:44.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:44 smithi114 ceph-mon[30312]: pgmap v1579: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:44.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:09:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:09:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:44.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:44.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:44 smithi114 ceph-mon[42466]: pgmap v1579: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:44 smithi119 ceph-mon[33714]: pgmap v1579: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-24T12:09:46.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:46 smithi114 ceph-mon[30312]: pgmap v1580: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:46 smithi114 ceph-mon[42466]: pgmap v1580: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:46.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:46 smithi119 ceph-mon[33714]: pgmap v1580: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:48.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:48 smithi114 ceph-mon[30312]: pgmap v1581: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:48 smithi114 ceph-mon[42466]: pgmap v1581: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:48 smithi119 ceph-mon[33714]: pgmap v1581: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:50.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:50 smithi114 ceph-mon[30312]: pgmap v1582: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:50.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:50 smithi114 ceph-mon[42466]: pgmap v1582: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:50 smithi119 ceph-mon[33714]: pgmap v1582: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:09:52.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:52 smithi114 ceph-mon[30312]: pgmap v1583: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:52 smithi114 ceph-mon[42466]: pgmap v1583: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:52.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:52 smithi119 ceph-mon[33714]: pgmap v1583: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:09:53.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:09:52 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:09:54.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:54.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:54 smithi114 ceph-mon[30312]: pgmap v1584: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:54.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:54.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:54 smithi114 ceph-mon[42466]: pgmap v1584: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:54.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:09:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:09:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:09:54.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:09:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:54 smithi119 ceph-mon[33714]: pgmap v1584: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:56.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:56 smithi114 ceph-mon[42466]: pgmap v1585: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:56.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:56 smithi114 ceph-mon[30312]: pgmap v1585: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:56.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:56 smithi119 ceph-mon[33714]: pgmap v1585: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:09:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:09:58 smithi119 ceph-mon[33714]: pgmap v1586: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:58.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:09:58 smithi114 ceph-mon[42466]: pgmap v1586: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:09:58.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:09:58 smithi114 ceph-mon[30312]: pgmap v1586: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:00 smithi119 ceph-mon[33714]: pgmap v1587: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:00 smithi119 ceph-mon[33714]: overall HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T12:10:00.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:00 smithi114 ceph-mon[42466]: pgmap v1587: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:00.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:00 smithi114 ceph-mon[42466]: overall HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T12:10:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:00 smithi114 ceph-mon[30312]: pgmap v1587: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:00 smithi114 ceph-mon[30312]: overall HEALTH_WARN 5 pool(s) do not have an application enabled 2024-03-24T12:10:02.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:02 smithi114 ceph-mon[30312]: pgmap v1588: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:02.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:02 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:10:02.789 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:02 smithi114 ceph-mon[42466]: pgmap v1588: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:02.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:02 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:10:02.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:02 smithi119 ceph-mon[33714]: pgmap v1588: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:02.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:02 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:10:03.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:10:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:10:04.720 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:04.721 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:04 smithi119 ceph-mon[33714]: pgmap v1589: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:04.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:04 smithi114 ceph-mon[30312]: pgmap v1589: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:04 smithi114 ceph-mon[42466]: pgmap v1589: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:10:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:10:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:05.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:05.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:05.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:06 smithi119 ceph-mon[33714]: pgmap v1590: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:10:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:10:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[30312]: pgmap v1590: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[42466]: pgmap v1590: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:06.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:10:06.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:10:06.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:10:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:08 smithi119 ceph-mon[33714]: pgmap v1591: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:08 smithi114 ceph-mon[30312]: pgmap v1591: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:08 smithi114 ceph-mon[42466]: pgmap v1591: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:10.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:10 smithi119 ceph-mon[33714]: pgmap v1592: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:10.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:10 smithi114 ceph-mon[30312]: pgmap v1592: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:10 smithi114 ceph-mon[42466]: pgmap v1592: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:12.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:12 smithi119 ceph-mon[33714]: pgmap v1593: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:12.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:12 smithi114 ceph-mon[30312]: pgmap v1593: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:12 smithi114 ceph-mon[42466]: pgmap v1593: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T12:10:13.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:10:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:10:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:14 smithi119 ceph-mon[33714]: pgmap v1594: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:14.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:14.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:14 smithi114 ceph-mon[30312]: pgmap v1594: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:14 smithi114 ceph-mon[42466]: pgmap v1594: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:10:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:10:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:16.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:16 smithi119 ceph-mon[33714]: pgmap v1595: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:16 smithi114 ceph-mon[30312]: pgmap v1595: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:16 smithi114 ceph-mon[42466]: pgmap v1595: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:18.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:18 smithi119 ceph-mon[33714]: pgmap v1596: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:18.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:18 smithi114 ceph-mon[30312]: pgmap v1596: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:18 smithi114 ceph-mon[42466]: pgmap v1596: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:20 smithi119 ceph-mon[33714]: pgmap v1597: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:20 smithi114 ceph-mon[30312]: pgmap v1597: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:20 smithi114 ceph-mon[42466]: pgmap v1597: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:22.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:22 smithi119 ceph-mon[33714]: pgmap v1598: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:10:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:22 smithi114 ceph-mon[30312]: pgmap v1598: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:10:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:22 smithi114 ceph-mon[42466]: pgmap v1598: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:10:23.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:10:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:10:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:24 smithi119 ceph-mon[33714]: pgmap v1599: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:24.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:24 smithi114 ceph-mon[30312]: pgmap v1599: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:24 smithi114 ceph-mon[42466]: pgmap v1599: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:10:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:10:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:26.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:26 smithi119 ceph-mon[33714]: pgmap v1600: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:26.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:26 smithi114 ceph-mon[30312]: pgmap v1600: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:26 smithi114 ceph-mon[42466]: pgmap v1600: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:28.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:28 smithi119 ceph-mon[33714]: pgmap v1601: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:28 smithi114 ceph-mon[30312]: pgmap v1601: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:28 smithi114 ceph-mon[42466]: pgmap v1601: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:30.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:30 smithi119 ceph-mon[33714]: pgmap v1602: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:30 smithi114 ceph-mon[30312]: pgmap v1602: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:30 smithi114 ceph-mon[42466]: pgmap v1602: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:32 smithi119 ceph-mon[33714]: pgmap v1603: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:32.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:32 smithi114 ceph-mon[42466]: pgmap v1603: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:32 smithi114 ceph-mon[30312]: pgmap v1603: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:33.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:10:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:10:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:34 smithi119 ceph-mon[33714]: pgmap v1604: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:34.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:10:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:10:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:34 smithi114 ceph-mon[42466]: pgmap v1604: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:34 smithi114 ceph-mon[30312]: pgmap v1604: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:35.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:35.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:35.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:36.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:36 smithi114 ceph-mon[42466]: pgmap v1605: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:36.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:36 smithi114 ceph-mon[30312]: pgmap v1605: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:36 smithi119 ceph-mon[33714]: pgmap v1605: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:38 smithi114 ceph-mon[30312]: pgmap v1606: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:38 smithi114 ceph-mon[42466]: pgmap v1606: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:39.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:38 smithi119 ceph-mon[33714]: pgmap v1606: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:40 smithi114 ceph-mon[30312]: pgmap v1607: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:40 smithi114 ceph-mon[42466]: pgmap v1607: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:41.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:40 smithi119 ceph-mon[33714]: pgmap v1607: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:42.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:42 smithi114 ceph-mon[42466]: pgmap v1608: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:42 smithi114 ceph-mon[30312]: pgmap v1608: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:43.020 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:42 smithi119 ceph-mon[33714]: pgmap v1608: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:43.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:10:43 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:10:44.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:44 smithi114 ceph-mon[42466]: pgmap v1609: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:44 smithi114 ceph-mon[30312]: pgmap v1609: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:10:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:10:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:44 smithi119 ceph-mon[33714]: pgmap v1609: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:46.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:46 smithi119 ceph-mon[33714]: pgmap v1610: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:46.803 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:10:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a[66430]: logger=cleanup t=2024-03-24T12:10:46.32716812Z level=info msg="Completed cleanup jobs" duration=1.132159ms 2024-03-24T12:10:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:46 smithi114 ceph-mon[30312]: pgmap v1610: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:46 smithi114 ceph-mon[42466]: pgmap v1610: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:48.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:48 smithi114 ceph-mon[30312]: pgmap v1611: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:48 smithi114 ceph-mon[42466]: pgmap v1611: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:49.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:48 smithi119 ceph-mon[33714]: pgmap v1611: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:50 smithi114 ceph-mon[30312]: pgmap v1612: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:50 smithi114 ceph-mon[42466]: pgmap v1612: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:50 smithi119 ceph-mon[33714]: pgmap v1612: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:10:52.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:52 smithi114 ceph-mon[30312]: pgmap v1613: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:52 smithi114 ceph-mon[42466]: pgmap v1613: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:53.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:52 smithi119 ceph-mon[33714]: pgmap v1613: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:53.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:10:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:10:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:54 smithi114 ceph-mon[30312]: pgmap v1614: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:10:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:10:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:10:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:54 smithi114 ceph-mon[42466]: pgmap v1614: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:55.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:10:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:54 smithi119 ceph-mon[33714]: pgmap v1614: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:56 smithi114 ceph-mon[30312]: pgmap v1615: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:56 smithi114 ceph-mon[42466]: pgmap v1615: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:57.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:56 smithi119 ceph-mon[33714]: pgmap v1615: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:10:58.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:10:58 smithi114 ceph-mon[30312]: pgmap v1616: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:10:58 smithi114 ceph-mon[42466]: pgmap v1616: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:10:59.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:10:58 smithi119 ceph-mon[33714]: pgmap v1616: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:00.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:00 smithi114 ceph-mon[30312]: pgmap v1617: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:00 smithi114 ceph-mon[42466]: pgmap v1617: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:01.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:00 smithi119 ceph-mon[33714]: pgmap v1617: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:02.855 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:02 smithi119 ceph-mon[33714]: pgmap v1618: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:11:02.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:02 smithi114 ceph-mon[30312]: pgmap v1618: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:11:02.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:02 smithi114 ceph-mon[42466]: pgmap v1618: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:11:03.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:11:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:11:04.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:04.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:04 smithi114 ceph-mon[30312]: pgmap v1619: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:04.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:04 smithi114 ceph-mon[42466]: pgmap v1619: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:04.990 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:11:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:11:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:04 smithi119 ceph-mon[33714]: pgmap v1619: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:05.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:06.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:06.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:06 smithi114 ceph-mon[30312]: pgmap v1620: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:06.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:11:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:06 smithi114 ceph-mon[42466]: pgmap v1620: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:11:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:06 smithi119 ceph-mon[33714]: pgmap v1620: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: pgmap v1621: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:11:08.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: pgmap v1621: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:11:08.739 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: pgmap v1621: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:08.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:11:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:11:08.804 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:11:10.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:10 smithi114 ceph-mon[30312]: pgmap v1622: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:10.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:10 smithi114 ceph-mon[42466]: pgmap v1622: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:10.808 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:10 smithi119 ceph-mon[33714]: pgmap v1622: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:12.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:12 smithi114 ceph-mon[30312]: pgmap v1623: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:12.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:12 smithi114 ceph-mon[42466]: pgmap v1623: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:12.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:12 smithi119 ceph-mon[33714]: pgmap v1623: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:13.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:11:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:11:14.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:14.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:14 smithi114 ceph-mon[30312]: pgmap v1624: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:14.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:14.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:14 smithi114 ceph-mon[42466]: pgmap v1624: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:14.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:11:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:11:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:14.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:14 smithi119 ceph-mon[33714]: pgmap v1624: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:16.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:16 smithi114 ceph-mon[30312]: pgmap v1625: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:16.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:16 smithi114 ceph-mon[42466]: pgmap v1625: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:16.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:16 smithi119 ceph-mon[33714]: pgmap v1625: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:18.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:18 smithi114 ceph-mon[30312]: pgmap v1626: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:18.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:18 smithi114 ceph-mon[42466]: pgmap v1626: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:18.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:18 smithi119 ceph-mon[33714]: pgmap v1626: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:20.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:20 smithi114 ceph-mon[30312]: pgmap v1627: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:20.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:20 smithi114 ceph-mon[42466]: pgmap v1627: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:20.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:20 smithi119 ceph-mon[33714]: pgmap v1627: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:22.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:22 smithi114 ceph-mon[30312]: pgmap v1628: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:22.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:22 smithi114 ceph-mon[42466]: pgmap v1628: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:22.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:22 smithi119 ceph-mon[33714]: pgmap v1628: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:23.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:11:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:11:24.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:24.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:24 smithi114 ceph-mon[30312]: pgmap v1629: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:24.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:11:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:11:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:24.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:24.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:24 smithi114 ceph-mon[42466]: pgmap v1629: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:24.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:24 smithi119 ceph-mon[33714]: pgmap v1629: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:26.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:26 smithi114 ceph-mon[30312]: pgmap v1630: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:26.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:26 smithi114 ceph-mon[42466]: pgmap v1630: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:26.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:26 smithi119 ceph-mon[33714]: pgmap v1630: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:28.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:28 smithi114 ceph-mon[30312]: pgmap v1631: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:28.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:28 smithi114 ceph-mon[42466]: pgmap v1631: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:28.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:28 smithi119 ceph-mon[33714]: pgmap v1631: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:30 smithi119 ceph-mon[33714]: pgmap v1632: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:30 smithi114 ceph-mon[30312]: pgmap v1632: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:30 smithi114 ceph-mon[42466]: pgmap v1632: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:32.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:32 smithi114 ceph-mon[30312]: pgmap v1633: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:32.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:32 smithi114 ceph-mon[42466]: pgmap v1633: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:32.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:32 smithi119 ceph-mon[33714]: pgmap v1633: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:33.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:11:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:11:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:34 smithi119 ceph-mon[33714]: pgmap v1634: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:34 smithi114 ceph-mon[30312]: pgmap v1634: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:34 smithi114 ceph-mon[42466]: pgmap v1634: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:11:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:11:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:35.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:35.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:35.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:36 smithi119 ceph-mon[33714]: pgmap v1635: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:36.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:36 smithi114 ceph-mon[30312]: pgmap v1635: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:36 smithi114 ceph-mon[42466]: pgmap v1635: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:38.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:38 smithi119 ceph-mon[33714]: pgmap v1636: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:38 smithi114 ceph-mon[30312]: pgmap v1636: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:38 smithi114 ceph-mon[42466]: pgmap v1636: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:40 smithi119 ceph-mon[33714]: pgmap v1637: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:40 smithi114 ceph-mon[30312]: pgmap v1637: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:40 smithi114 ceph-mon[42466]: pgmap v1637: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:42.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:42 smithi119 ceph-mon[33714]: pgmap v1638: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:42 smithi114 ceph-mon[30312]: pgmap v1638: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:42 smithi114 ceph-mon[42466]: pgmap v1638: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:43.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:11:43 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:11:44.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:44 smithi119 ceph-mon[33714]: pgmap v1639: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:44.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:44.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:44 smithi114 ceph-mon[42466]: pgmap v1639: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:44 smithi114 ceph-mon[30312]: pgmap v1639: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:11:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:11:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:46.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:46 smithi119 ceph-mon[33714]: pgmap v1640: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:46 smithi114 ceph-mon[30312]: pgmap v1640: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:46 smithi114 ceph-mon[42466]: pgmap v1640: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:48.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:48 smithi119 ceph-mon[33714]: pgmap v1641: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:48.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:48 smithi114 ceph-mon[30312]: pgmap v1641: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:48 smithi114 ceph-mon[42466]: pgmap v1641: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:50.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:50 smithi119 ceph-mon[33714]: pgmap v1642: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:50.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:50 smithi114 ceph-mon[30312]: pgmap v1642: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:50 smithi114 ceph-mon[42466]: pgmap v1642: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:11:52.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:52 smithi119 ceph-mon[33714]: pgmap v1643: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:52.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:52 smithi114 ceph-mon[30312]: pgmap v1643: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:52 smithi114 ceph-mon[42466]: pgmap v1643: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:53.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:11:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:11:54.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:54 smithi119 ceph-mon[33714]: pgmap v1644: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:54 smithi114 ceph-mon[30312]: pgmap v1644: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:11:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:54 smithi114 ceph-mon[42466]: pgmap v1644: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:11:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:11:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:11:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:56 smithi119 ceph-mon[33714]: pgmap v1645: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:56 smithi114 ceph-mon[42466]: pgmap v1645: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:56.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:56 smithi114 ceph-mon[30312]: pgmap v1645: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:11:58.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:11:58 smithi119 ceph-mon[33714]: pgmap v1646: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:58.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:11:58 smithi114 ceph-mon[42466]: pgmap v1646: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:11:58.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:11:58 smithi114 ceph-mon[30312]: pgmap v1646: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:00.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:00 smithi119 ceph-mon[33714]: pgmap v1647: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:00.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:00 smithi114 ceph-mon[42466]: pgmap v1647: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:00.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:00 smithi114 ceph-mon[30312]: pgmap v1647: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:02.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:02 smithi114 ceph-mon[30312]: pgmap v1648: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:02.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:02 smithi114 ceph-mon[42466]: pgmap v1648: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:02.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:02 smithi119 ceph-mon[33714]: pgmap v1648: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:03.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:12:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:12:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:04 smithi119 ceph-mon[33714]: pgmap v1649: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:04.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:04.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:04 smithi114 ceph-mon[30312]: pgmap v1649: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:04 smithi114 ceph-mon[42466]: pgmap v1649: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:12:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:12:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:05.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:05.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:06.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:06 smithi119 ceph-mon[33714]: pgmap v1650: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:06.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:06 smithi114 ceph-mon[30312]: pgmap v1650: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:06 smithi114 ceph-mon[42466]: pgmap v1650: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:07.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:07 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:12:07.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:07 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:12:08.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:07 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:12:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[30312]: pgmap v1651: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:12:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:12:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:12:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[42466]: pgmap v1651: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:12:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:12:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:08 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:12:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:08 smithi119 ceph-mon[33714]: pgmap v1651: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:12:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:12:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:08 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:12:10.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:10 smithi114 ceph-mon[30312]: pgmap v1652: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:10 smithi114 ceph-mon[42466]: pgmap v1652: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:11.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:10 smithi119 ceph-mon[33714]: pgmap v1652: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:12.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:12 smithi114 ceph-mon[30312]: pgmap v1653: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:12 smithi114 ceph-mon[42466]: pgmap v1653: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:13.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:12 smithi119 ceph-mon[33714]: pgmap v1653: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:13.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:12:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:12:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:14 smithi114 ceph-mon[30312]: pgmap v1654: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:12:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:12:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:14 smithi114 ceph-mon[42466]: pgmap v1654: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:15.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:14 smithi119 ceph-mon[33714]: pgmap v1654: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:16.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:16 smithi114 ceph-mon[30312]: pgmap v1655: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:16 smithi114 ceph-mon[42466]: pgmap v1655: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:16 smithi119 ceph-mon[33714]: pgmap v1655: 292 pgs: 292 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:12:17.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:17 smithi114 ceph-mon[30312]: osdmap e755: 8 total, 8 up, 8 in 2024-03-24T12:12:17.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:17 smithi114 ceph-mon[42466]: osdmap e755: 8 total, 8 up, 8 in 2024-03-24T12:12:18.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:17 smithi119 ceph-mon[33714]: osdmap e755: 8 total, 8 up, 8 in 2024-03-24T12:12:18.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:18 smithi114 ceph-mon[42466]: pgmap v1657: 228 pgs: 228 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:12:18.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:18 smithi114 ceph-mon[42466]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:18.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:18 smithi114 ceph-mon[42466]: osdmap e756: 8 total, 8 up, 8 in 2024-03-24T12:12:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:18 smithi114 ceph-mon[30312]: pgmap v1657: 228 pgs: 228 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:12:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:18 smithi114 ceph-mon[30312]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:18 smithi114 ceph-mon[30312]: osdmap e756: 8 total, 8 up, 8 in 2024-03-24T12:12:19.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:18 smithi119 ceph-mon[33714]: pgmap v1657: 228 pgs: 228 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:12:19.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:18 smithi119 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:19.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:18 smithi119 ceph-mon[33714]: osdmap e756: 8 total, 8 up, 8 in 2024-03-24T12:12:19.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:19 smithi114 ceph-mon[30312]: osdmap e757: 8 total, 8 up, 8 in 2024-03-24T12:12:19.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:19 smithi114 ceph-mon[42466]: osdmap e757: 8 total, 8 up, 8 in 2024-03-24T12:12:20.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:19 smithi119 ceph-mon[33714]: osdmap e757: 8 total, 8 up, 8 in 2024-03-24T12:12:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:20 smithi114 ceph-mon[30312]: pgmap v1660: 292 pgs: 96 unknown, 196 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:12:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:20 smithi114 ceph-mon[30312]: osdmap e758: 8 total, 8 up, 8 in 2024-03-24T12:12:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:20 smithi114 ceph-mon[42466]: pgmap v1660: 292 pgs: 96 unknown, 196 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:12:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:20 smithi114 ceph-mon[42466]: osdmap e758: 8 total, 8 up, 8 in 2024-03-24T12:12:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:20 smithi119 ceph-mon[33714]: pgmap v1660: 292 pgs: 96 unknown, 196 active+clean; 582 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:12:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:20 smithi119 ceph-mon[33714]: osdmap e758: 8 total, 8 up, 8 in 2024-03-24T12:12:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:21.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:21 smithi114 ceph-mon[30312]: osdmap e759: 8 total, 8 up, 8 in 2024-03-24T12:12:21.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:21 smithi114 ceph-mon[42466]: osdmap e759: 8 total, 8 up, 8 in 2024-03-24T12:12:22.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:21 smithi119 ceph-mon[33714]: osdmap e759: 8 total, 8 up, 8 in 2024-03-24T12:12:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:22 smithi114 ceph-mon[30312]: pgmap v1663: 292 pgs: 96 creating+peering, 196 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:22 smithi114 ceph-mon[30312]: osdmap e760: 8 total, 8 up, 8 in 2024-03-24T12:12:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:22 smithi114 ceph-mon[42466]: pgmap v1663: 292 pgs: 96 creating+peering, 196 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:22 smithi114 ceph-mon[42466]: osdmap e760: 8 total, 8 up, 8 in 2024-03-24T12:12:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:22 smithi119 ceph-mon[33714]: pgmap v1663: 292 pgs: 96 creating+peering, 196 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:22 smithi119 ceph-mon[33714]: osdmap e760: 8 total, 8 up, 8 in 2024-03-24T12:12:23.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:12:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:12:23.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:23 smithi114 ceph-mon[30312]: osdmap e761: 8 total, 8 up, 8 in 2024-03-24T12:12:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:23 smithi114 ceph-mon[30312]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:23 smithi114 ceph-mon[42466]: osdmap e761: 8 total, 8 up, 8 in 2024-03-24T12:12:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:23 smithi114 ceph-mon[42466]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:23 smithi119 ceph-mon[33714]: osdmap e761: 8 total, 8 up, 8 in 2024-03-24T12:12:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:23 smithi119 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:24.698 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: handle_notify cookie 94717734204144 notify_id 3242700308481 notifier_gid 24895 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (1809369 ms) 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: Trying... 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: handle_notify cookie 94717746927728 notify_id 3255585210370 notifier_gid 56084 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: Waiting for 3.000000000s 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: Timed out. 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: Flushing... 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: Flushed... 2024-03-24T12:12:24.699 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7057 ms) 2024-03-24T12:12:24.700 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (1816426 ms total) 2024-03-24T12:12:24.700 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: 2024-03-24T12:12:24.700 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-24T12:12:24.700 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (1816426 ms total) 2024-03-24T12:12:24.700 INFO:tasks.workunit.client.0.smithi114.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-24T12:12:24.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:24.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:24 smithi114 ceph-mon[42466]: pgmap v1666: 228 pgs: 32 creating+peering, 196 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:24.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:24 smithi114 ceph-mon[42466]: osdmap e762: 8 total, 8 up, 8 in 2024-03-24T12:12:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:24 smithi114 ceph-mon[30312]: pgmap v1666: 228 pgs: 32 creating+peering, 196 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:24 smithi114 ceph-mon[30312]: osdmap e762: 8 total, 8 up, 8 in 2024-03-24T12:12:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:12:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:12:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:24 smithi119 ceph-mon[33714]: pgmap v1666: 228 pgs: 32 creating+peering, 196 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:24 smithi119 ceph-mon[33714]: osdmap e762: 8 total, 8 up, 8 in 2024-03-24T12:12:25.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:25 smithi114 ceph-mon[30312]: osdmap e763: 8 total, 8 up, 8 in 2024-03-24T12:12:25.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:25 smithi114 ceph-mon[42466]: osdmap e763: 8 total, 8 up, 8 in 2024-03-24T12:12:26.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:25 smithi119 ceph-mon[33714]: osdmap e763: 8 total, 8 up, 8 in 2024-03-24T12:12:26.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:26 smithi114 ceph-mon[42466]: pgmap v1669: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:12:26.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:26 smithi114 ceph-mon[42466]: osdmap e764: 8 total, 8 up, 8 in 2024-03-24T12:12:26.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:26 smithi114 ceph-mon[30312]: pgmap v1669: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:12:26.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:26 smithi114 ceph-mon[30312]: osdmap e764: 8 total, 8 up, 8 in 2024-03-24T12:12:27.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:26 smithi119 ceph-mon[33714]: pgmap v1669: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:12:27.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:26 smithi119 ceph-mon[33714]: osdmap e764: 8 total, 8 up, 8 in 2024-03-24T12:12:27.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:27 smithi114 ceph-mon[30312]: osdmap e765: 8 total, 8 up, 8 in 2024-03-24T12:12:27.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:27 smithi114 ceph-mon[42466]: osdmap e765: 8 total, 8 up, 8 in 2024-03-24T12:12:28.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:27 smithi119 ceph-mon[33714]: osdmap e765: 8 total, 8 up, 8 in 2024-03-24T12:12:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:28 smithi114 ceph-mon[30312]: pgmap v1672: 260 pgs: 64 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:12:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:28 smithi114 ceph-mon[30312]: osdmap e766: 8 total, 8 up, 8 in 2024-03-24T12:12:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:28 smithi114 ceph-mon[30312]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:28 smithi114 ceph-mon[42466]: pgmap v1672: 260 pgs: 64 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:12:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:28 smithi114 ceph-mon[42466]: osdmap e766: 8 total, 8 up, 8 in 2024-03-24T12:12:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:28 smithi114 ceph-mon[42466]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:28 smithi119 ceph-mon[33714]: pgmap v1672: 260 pgs: 64 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:12:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:28 smithi119 ceph-mon[33714]: osdmap e766: 8 total, 8 up, 8 in 2024-03-24T12:12:29.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:28 smithi119 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:30.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:29 smithi119 ceph-mon[33714]: osdmap e767: 8 total, 8 up, 8 in 2024-03-24T12:12:30.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:29 smithi114 ceph-mon[30312]: osdmap e767: 8 total, 8 up, 8 in 2024-03-24T12:12:30.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:29 smithi114 ceph-mon[42466]: osdmap e767: 8 total, 8 up, 8 in 2024-03-24T12:12:31.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:30 smithi119 ceph-mon[33714]: pgmap v1675: 164 pgs: 164 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:30 smithi119 ceph-mon[33714]: osdmap e768: 8 total, 8 up, 8 in 2024-03-24T12:12:31.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:30 smithi114 ceph-mon[30312]: pgmap v1675: 164 pgs: 164 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:31.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:30 smithi114 ceph-mon[30312]: osdmap e768: 8 total, 8 up, 8 in 2024-03-24T12:12:31.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:30 smithi114 ceph-mon[42466]: pgmap v1675: 164 pgs: 164 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:31.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:30 smithi114 ceph-mon[42466]: osdmap e768: 8 total, 8 up, 8 in 2024-03-24T12:12:32.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:31 smithi119 ceph-mon[33714]: osdmap e769: 8 total, 8 up, 8 in 2024-03-24T12:12:32.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:31 smithi119 ceph-mon[33714]: pgmap v1678: 260 pgs: 260 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T12:12:32.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:31 smithi114 ceph-mon[42466]: osdmap e769: 8 total, 8 up, 8 in 2024-03-24T12:12:32.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:31 smithi114 ceph-mon[42466]: pgmap v1678: 260 pgs: 260 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T12:12:32.139 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:31 smithi114 ceph-mon[30312]: osdmap e769: 8 total, 8 up, 8 in 2024-03-24T12:12:32.139 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:31 smithi114 ceph-mon[30312]: pgmap v1678: 260 pgs: 260 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T12:12:33.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:32 smithi119 ceph-mon[33714]: osdmap e770: 8 total, 8 up, 8 in 2024-03-24T12:12:33.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:32 smithi114 ceph-mon[30312]: osdmap e770: 8 total, 8 up, 8 in 2024-03-24T12:12:33.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:32 smithi114 ceph-mon[42466]: osdmap e770: 8 total, 8 up, 8 in 2024-03-24T12:12:33.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:12:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:12:34.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:33 smithi119 ceph-mon[33714]: osdmap e771: 8 total, 8 up, 8 in 2024-03-24T12:12:34.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:33 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:34.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:33 smithi119 ceph-mon[33714]: pgmap v1681: 260 pgs: 64 unknown, 196 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:34.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:33 smithi114 ceph-mon[30312]: osdmap e771: 8 total, 8 up, 8 in 2024-03-24T12:12:34.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:33 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:34.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:33 smithi114 ceph-mon[30312]: pgmap v1681: 260 pgs: 64 unknown, 196 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:34.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:33 smithi114 ceph-mon[42466]: osdmap e771: 8 total, 8 up, 8 in 2024-03-24T12:12:34.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:33 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:34.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:33 smithi114 ceph-mon[42466]: pgmap v1681: 260 pgs: 64 unknown, 196 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: Running main() from gmock_main.cc 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [----------] Global test environment set-up. 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (1808349 ms) 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-24T12:12:34.800 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3025 ms) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3020 ms) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3034 ms) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3021 ms) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3043 ms) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (3021 ms) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (1826513 ms total) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [----------] Global test environment tear-down 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (1826513 ms total) 2024-03-24T12:12:34.801 INFO:tasks.workunit.client.0.smithi114.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-24T12:12:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:34 smithi114 ceph-mon[30312]: osdmap e772: 8 total, 8 up, 8 in 2024-03-24T12:12:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:34 smithi114 ceph-mon[42466]: osdmap e772: 8 total, 8 up, 8 in 2024-03-24T12:12:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:12:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:12:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:35.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:34 smithi119 ceph-mon[33714]: osdmap e772: 8 total, 8 up, 8 in 2024-03-24T12:12:36.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:35 smithi119 ceph-mon[33714]: osdmap e773: 8 total, 8 up, 8 in 2024-03-24T12:12:36.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:36.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:35 smithi119 ceph-mon[33714]: pgmap v1684: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:36.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:35 smithi114 ceph-mon[30312]: osdmap e773: 8 total, 8 up, 8 in 2024-03-24T12:12:36.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:36.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:35 smithi114 ceph-mon[30312]: pgmap v1684: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:36.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:35 smithi114 ceph-mon[42466]: osdmap e773: 8 total, 8 up, 8 in 2024-03-24T12:12:36.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:36.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:35 smithi114 ceph-mon[42466]: pgmap v1684: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 736 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:36 smithi119 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:36 smithi119 ceph-mon[33714]: osdmap e774: 8 total, 8 up, 8 in 2024-03-24T12:12:37.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:36 smithi114 ceph-mon[30312]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:37.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:36 smithi114 ceph-mon[30312]: osdmap e774: 8 total, 8 up, 8 in 2024-03-24T12:12:37.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:36 smithi114 ceph-mon[42466]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:37.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:36 smithi114 ceph-mon[42466]: osdmap e774: 8 total, 8 up, 8 in 2024-03-24T12:12:38.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:37 smithi114 ceph-mon[30312]: osdmap e775: 8 total, 8 up, 8 in 2024-03-24T12:12:38.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:37 smithi114 ceph-mon[30312]: pgmap v1687: 228 pgs: 32 creating+peering, 196 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T12:12:38.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:37 smithi114 ceph-mon[42466]: osdmap e775: 8 total, 8 up, 8 in 2024-03-24T12:12:38.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:37 smithi114 ceph-mon[42466]: pgmap v1687: 228 pgs: 32 creating+peering, 196 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T12:12:38.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:37 smithi119 ceph-mon[33714]: osdmap e775: 8 total, 8 up, 8 in 2024-03-24T12:12:38.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:37 smithi119 ceph-mon[33714]: pgmap v1687: 228 pgs: 32 creating+peering, 196 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T12:12:39.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:38 smithi114 ceph-mon[30312]: osdmap e776: 8 total, 8 up, 8 in 2024-03-24T12:12:39.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:38 smithi114 ceph-mon[42466]: osdmap e776: 8 total, 8 up, 8 in 2024-03-24T12:12:39.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:38 smithi119 ceph-mon[33714]: osdmap e776: 8 total, 8 up, 8 in 2024-03-24T12:12:40.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:39 smithi114 ceph-mon[30312]: osdmap e777: 8 total, 8 up, 8 in 2024-03-24T12:12:40.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:39 smithi114 ceph-mon[30312]: pgmap v1690: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T12:12:40.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:39 smithi114 ceph-mon[42466]: osdmap e777: 8 total, 8 up, 8 in 2024-03-24T12:12:40.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:39 smithi114 ceph-mon[42466]: pgmap v1690: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T12:12:40.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:39 smithi119 ceph-mon[33714]: osdmap e777: 8 total, 8 up, 8 in 2024-03-24T12:12:40.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:39 smithi119 ceph-mon[33714]: pgmap v1690: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 771 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-24T12:12:41.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:40 smithi114 ceph-mon[30312]: osdmap e778: 8 total, 8 up, 8 in 2024-03-24T12:12:41.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:40 smithi114 ceph-mon[42466]: osdmap e778: 8 total, 8 up, 8 in 2024-03-24T12:12:41.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:40 smithi119 ceph-mon[33714]: osdmap e778: 8 total, 8 up, 8 in 2024-03-24T12:12:42.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:41 smithi114 ceph-mon[30312]: osdmap e779: 8 total, 8 up, 8 in 2024-03-24T12:12:42.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:41 smithi114 ceph-mon[30312]: pgmap v1693: 164 pgs: 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:42.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:41 smithi114 ceph-mon[42466]: osdmap e779: 8 total, 8 up, 8 in 2024-03-24T12:12:42.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:41 smithi114 ceph-mon[42466]: pgmap v1693: 164 pgs: 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:42.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:41 smithi119 ceph-mon[33714]: osdmap e779: 8 total, 8 up, 8 in 2024-03-24T12:12:42.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:41 smithi119 ceph-mon[33714]: pgmap v1693: 164 pgs: 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:43.136 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:42 smithi119 ceph-mon[33714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:43.137 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:42 smithi119 ceph-mon[33714]: osdmap e780: 8 total, 8 up, 8 in 2024-03-24T12:12:43.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:42 smithi114 ceph-mon[30312]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:43.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:42 smithi114 ceph-mon[30312]: osdmap e780: 8 total, 8 up, 8 in 2024-03-24T12:12:43.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:42 smithi114 ceph-mon[42466]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:43.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:42 smithi114 ceph-mon[42466]: osdmap e780: 8 total, 8 up, 8 in 2024-03-24T12:12:43.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:12:43 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:12:44.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:43 smithi114 ceph-mon[30312]: osdmap e781: 8 total, 8 up, 8 in 2024-03-24T12:12:44.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:43 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:44.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:43 smithi114 ceph-mon[30312]: pgmap v1696: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:43 smithi114 ceph-mon[42466]: osdmap e781: 8 total, 8 up, 8 in 2024-03-24T12:12:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:43 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:43 smithi114 ceph-mon[42466]: pgmap v1696: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:44.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:43 smithi119 ceph-mon[33714]: osdmap e781: 8 total, 8 up, 8 in 2024-03-24T12:12:44.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:43 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:44.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:43 smithi119 ceph-mon[33714]: pgmap v1696: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:44.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:44 smithi114 ceph-mon[30312]: osdmap e782: 8 total, 8 up, 8 in 2024-03-24T12:12:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:12:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:12:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:44 smithi114 ceph-mon[42466]: osdmap e782: 8 total, 8 up, 8 in 2024-03-24T12:12:45.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:44 smithi119 ceph-mon[33714]: osdmap e782: 8 total, 8 up, 8 in 2024-03-24T12:12:46.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:45 smithi114 ceph-mon[30312]: osdmap e783: 8 total, 8 up, 8 in 2024-03-24T12:12:46.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:45 smithi114 ceph-mon[30312]: pgmap v1699: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:46.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:45 smithi114 ceph-mon[42466]: osdmap e783: 8 total, 8 up, 8 in 2024-03-24T12:12:46.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:45 smithi114 ceph-mon[42466]: pgmap v1699: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:46.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:45 smithi119 ceph-mon[33714]: osdmap e783: 8 total, 8 up, 8 in 2024-03-24T12:12:46.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:45 smithi119 ceph-mon[33714]: pgmap v1699: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 777 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:47.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:46 smithi114 ceph-mon[30312]: osdmap e784: 8 total, 8 up, 8 in 2024-03-24T12:12:47.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:46 smithi114 ceph-mon[42466]: osdmap e784: 8 total, 8 up, 8 in 2024-03-24T12:12:47.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:46 smithi119 ceph-mon[33714]: osdmap e784: 8 total, 8 up, 8 in 2024-03-24T12:12:48.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:47 smithi114 ceph-mon[30312]: osdmap e785: 8 total, 8 up, 8 in 2024-03-24T12:12:48.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:47 smithi114 ceph-mon[30312]: pgmap v1702: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:48.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:47 smithi114 ceph-mon[42466]: osdmap e785: 8 total, 8 up, 8 in 2024-03-24T12:12:48.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:47 smithi114 ceph-mon[42466]: pgmap v1702: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:48.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:47 smithi119 ceph-mon[33714]: osdmap e785: 8 total, 8 up, 8 in 2024-03-24T12:12:48.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:47 smithi119 ceph-mon[33714]: pgmap v1702: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:49.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:48 smithi114 ceph-mon[42466]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:49.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:48 smithi114 ceph-mon[42466]: osdmap e786: 8 total, 8 up, 8 in 2024-03-24T12:12:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:48 smithi114 ceph-mon[30312]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:49.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:48 smithi114 ceph-mon[30312]: osdmap e786: 8 total, 8 up, 8 in 2024-03-24T12:12:49.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:48 smithi119 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:49.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:48 smithi119 ceph-mon[33714]: osdmap e786: 8 total, 8 up, 8 in 2024-03-24T12:12:50.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:49 smithi114 ceph-mon[30312]: osdmap e787: 8 total, 8 up, 8 in 2024-03-24T12:12:50.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:49 smithi114 ceph-mon[30312]: pgmap v1705: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:50.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:49 smithi114 ceph-mon[42466]: osdmap e787: 8 total, 8 up, 8 in 2024-03-24T12:12:50.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:49 smithi114 ceph-mon[42466]: pgmap v1705: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:50.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:49 smithi119 ceph-mon[33714]: osdmap e787: 8 total, 8 up, 8 in 2024-03-24T12:12:50.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:49 smithi119 ceph-mon[33714]: pgmap v1705: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:51.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:50 smithi114 ceph-mon[30312]: osdmap e788: 8 total, 8 up, 8 in 2024-03-24T12:12:51.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:51.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:50 smithi114 ceph-mon[42466]: osdmap e788: 8 total, 8 up, 8 in 2024-03-24T12:12:51.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:51.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:50 smithi119 ceph-mon[33714]: osdmap e788: 8 total, 8 up, 8 in 2024-03-24T12:12:51.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:12:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:51 smithi114 ceph-mon[30312]: osdmap e789: 8 total, 8 up, 8 in 2024-03-24T12:12:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:51 smithi114 ceph-mon[30312]: pgmap v1708: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T12:12:52.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:51 smithi114 ceph-mon[30312]: osdmap e790: 8 total, 8 up, 8 in 2024-03-24T12:12:52.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:51 smithi114 ceph-mon[42466]: osdmap e789: 8 total, 8 up, 8 in 2024-03-24T12:12:52.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:51 smithi114 ceph-mon[42466]: pgmap v1708: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T12:12:52.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:51 smithi114 ceph-mon[42466]: osdmap e790: 8 total, 8 up, 8 in 2024-03-24T12:12:52.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:51 smithi119 ceph-mon[33714]: osdmap e789: 8 total, 8 up, 8 in 2024-03-24T12:12:52.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:51 smithi119 ceph-mon[33714]: pgmap v1708: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T12:12:52.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:51 smithi119 ceph-mon[33714]: osdmap e790: 8 total, 8 up, 8 in 2024-03-24T12:12:53.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:12:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:12:54.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[30312]: osdmap e791: 8 total, 8 up, 8 in 2024-03-24T12:12:54.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:54.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[30312]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:54.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[30312]: pgmap v1711: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:54.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[42466]: osdmap e791: 8 total, 8 up, 8 in 2024-03-24T12:12:54.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:54.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[42466]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:54.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:53 smithi114 ceph-mon[42466]: pgmap v1711: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:54.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:53 smithi119 ceph-mon[33714]: osdmap e791: 8 total, 8 up, 8 in 2024-03-24T12:12:54.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:53 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:12:54.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:53 smithi119 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:54.306 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:53 smithi119 ceph-mon[33714]: pgmap v1711: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:12:54.933 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:12:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:12:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:12:55.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:54 smithi114 ceph-mon[30312]: osdmap e792: 8 total, 8 up, 8 in 2024-03-24T12:12:55.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:54 smithi114 ceph-mon[42466]: osdmap e792: 8 total, 8 up, 8 in 2024-03-24T12:12:55.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:54 smithi119 ceph-mon[33714]: osdmap e792: 8 total, 8 up, 8 in 2024-03-24T12:12:55.953 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: Running main() from gmock_main.cc 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [----------] Global test environment set-up. 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (1808363 ms) 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3018 ms) 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3038 ms) 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3023 ms) 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-24T12:12:55.954 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3023 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3048 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3018 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3028 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3019 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3013 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3018 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3025 ms) 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-24T12:12:55.955 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3029 ms) 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3015 ms) 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (1847678 ms total) 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [----------] Global test environment tear-down 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (1847678 ms total) 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82759 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82759 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83237 2024-03-24T12:12:55.956 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83237 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83677 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83677 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82705 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82705 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83052 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83052 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83496 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83496 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83199 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83199 2024-03-24T12:12:55.957 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83565 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83565 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82988 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82988 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82543 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82543 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83644 2024-03-24T12:12:55.958 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83644 2024-03-24T12:12:56.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:55 smithi114 ceph-mon[30312]: osdmap e793: 8 total, 8 up, 8 in 2024-03-24T12:12:56.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:55 smithi114 ceph-mon[30312]: pgmap v1714: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 821 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:56.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:55 smithi114 ceph-mon[42466]: osdmap e793: 8 total, 8 up, 8 in 2024-03-24T12:12:56.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:55 smithi114 ceph-mon[42466]: pgmap v1714: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 821 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:56.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:55 smithi119 ceph-mon[33714]: osdmap e793: 8 total, 8 up, 8 in 2024-03-24T12:12:56.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:55 smithi119 ceph-mon[33714]: pgmap v1714: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 821 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:12:57.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:56 smithi114 ceph-mon[30312]: osdmap e794: 8 total, 8 up, 8 in 2024-03-24T12:12:57.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:56 smithi114 ceph-mon[42466]: osdmap e794: 8 total, 8 up, 8 in 2024-03-24T12:12:57.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:56 smithi119 ceph-mon[33714]: osdmap e794: 8 total, 8 up, 8 in 2024-03-24T12:12:58.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:57 smithi114 ceph-mon[30312]: osdmap e795: 8 total, 8 up, 8 in 2024-03-24T12:12:58.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:57 smithi114 ceph-mon[30312]: pgmap v1717: 196 pgs: 196 active+clean; 585 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-24T12:12:58.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:57 smithi114 ceph-mon[42466]: osdmap e795: 8 total, 8 up, 8 in 2024-03-24T12:12:58.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:57 smithi114 ceph-mon[42466]: pgmap v1717: 196 pgs: 196 active+clean; 585 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-24T12:12:58.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:57 smithi119 ceph-mon[33714]: osdmap e795: 8 total, 8 up, 8 in 2024-03-24T12:12:58.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:57 smithi119 ceph-mon[33714]: pgmap v1717: 196 pgs: 196 active+clean; 585 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-24T12:12:59.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:58 smithi114 ceph-mon[30312]: osdmap e796: 8 total, 8 up, 8 in 2024-03-24T12:12:59.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:58 smithi114 ceph-mon[30312]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:59.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:58 smithi114 ceph-mon[42466]: osdmap e796: 8 total, 8 up, 8 in 2024-03-24T12:12:59.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:58 smithi114 ceph-mon[42466]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:58 smithi119 ceph-mon[33714]: osdmap e796: 8 total, 8 up, 8 in 2024-03-24T12:12:59.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:58 smithi119 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:13:00.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:59 smithi114 ceph-mon[42466]: osdmap e797: 8 total, 8 up, 8 in 2024-03-24T12:13:00.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:12:59 smithi114 ceph-mon[42466]: pgmap v1720: 196 pgs: 196 active+clean; 585 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-24T12:13:00.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:59 smithi114 ceph-mon[30312]: osdmap e797: 8 total, 8 up, 8 in 2024-03-24T12:13:00.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:12:59 smithi114 ceph-mon[30312]: pgmap v1720: 196 pgs: 196 active+clean; 585 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-24T12:13:00.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:59 smithi119 ceph-mon[33714]: osdmap e797: 8 total, 8 up, 8 in 2024-03-24T12:13:00.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:12:59 smithi119 ceph-mon[33714]: pgmap v1720: 196 pgs: 196 active+clean; 585 KiB data, 857 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-24T12:13:01.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:00 smithi114 ceph-mon[30312]: osdmap e798: 8 total, 8 up, 8 in 2024-03-24T12:13:01.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:00 smithi114 ceph-mon[42466]: osdmap e798: 8 total, 8 up, 8 in 2024-03-24T12:13:01.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:00 smithi119 ceph-mon[33714]: osdmap e798: 8 total, 8 up, 8 in 2024-03-24T12:13:02.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:01 smithi119 ceph-mon[33714]: osdmap e799: 8 total, 8 up, 8 in 2024-03-24T12:13:02.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:01 smithi119 ceph-mon[33714]: pgmap v1723: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:13:02.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:01 smithi114 ceph-mon[30312]: osdmap e799: 8 total, 8 up, 8 in 2024-03-24T12:13:02.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:01 smithi114 ceph-mon[30312]: pgmap v1723: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:13:02.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:01 smithi114 ceph-mon[42466]: osdmap e799: 8 total, 8 up, 8 in 2024-03-24T12:13:02.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:01 smithi114 ceph-mon[42466]: pgmap v1723: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:13:03.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:03 smithi119 ceph-mon[33714]: osdmap e800: 8 total, 8 up, 8 in 2024-03-24T12:13:03.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:13:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:13:03.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:03 smithi114 ceph-mon[30312]: osdmap e800: 8 total, 8 up, 8 in 2024-03-24T12:13:03.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:03 smithi114 ceph-mon[42466]: osdmap e800: 8 total, 8 up, 8 in 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:04 smithi119 ceph-mon[33714]: osdmap e801: 8 total, 8 up, 8 in 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:04.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:04 smithi119 ceph-mon[33714]: pgmap v1726: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:13:04.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:04 smithi114 ceph-mon[42466]: osdmap e801: 8 total, 8 up, 8 in 2024-03-24T12:13:04.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:04.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:04 smithi114 ceph-mon[42466]: pgmap v1726: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:13:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:04 smithi114 ceph-mon[30312]: osdmap e801: 8 total, 8 up, 8 in 2024-03-24T12:13:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:04.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:04 smithi114 ceph-mon[30312]: pgmap v1726: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-24T12:13:04.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:13:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:13:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:05.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:05 smithi119 ceph-mon[33714]: osdmap e802: 8 total, 8 up, 8 in 2024-03-24T12:13:05.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:05 smithi114 ceph-mon[30312]: osdmap e802: 8 total, 8 up, 8 in 2024-03-24T12:13:05.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:05 smithi114 ceph-mon[42466]: osdmap e802: 8 total, 8 up, 8 in 2024-03-24T12:13:06.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:06 smithi119 ceph-mon[33714]: osdmap e803: 8 total, 8 up, 8 in 2024-03-24T12:13:06.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:06 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:06.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:06 smithi119 ceph-mon[33714]: pgmap v1729: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:13:06.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:06 smithi114 ceph-mon[30312]: osdmap e803: 8 total, 8 up, 8 in 2024-03-24T12:13:06.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:06 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:06.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:06 smithi114 ceph-mon[30312]: pgmap v1729: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:13:06.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:06 smithi114 ceph-mon[42466]: osdmap e803: 8 total, 8 up, 8 in 2024-03-24T12:13:06.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:06 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:06.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:06 smithi114 ceph-mon[42466]: pgmap v1729: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:13:07.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:07 smithi119 ceph-mon[33714]: osdmap e804: 8 total, 8 up, 8 in 2024-03-24T12:13:07.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:07 smithi114 ceph-mon[30312]: osdmap e804: 8 total, 8 up, 8 in 2024-03-24T12:13:07.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:07 smithi114 ceph-mon[42466]: osdmap e804: 8 total, 8 up, 8 in 2024-03-24T12:13:08.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:08 smithi119 ceph-mon[33714]: osdmap e805: 8 total, 8 up, 8 in 2024-03-24T12:13:08.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:08 smithi119 ceph-mon[33714]: pgmap v1732: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:08.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:08 smithi114 ceph-mon[30312]: osdmap e805: 8 total, 8 up, 8 in 2024-03-24T12:13:08.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:08 smithi114 ceph-mon[30312]: pgmap v1732: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:08.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:08 smithi114 ceph-mon[42466]: osdmap e805: 8 total, 8 up, 8 in 2024-03-24T12:13:08.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:08 smithi114 ceph-mon[42466]: pgmap v1732: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:09.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:09 smithi119 ceph-mon[33714]: osdmap e806: 8 total, 8 up, 8 in 2024-03-24T12:13:09.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:09 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:13:09.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:09 smithi114 ceph-mon[30312]: osdmap e806: 8 total, 8 up, 8 in 2024-03-24T12:13:09.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:09 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:13:09.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:09 smithi114 ceph-mon[42466]: osdmap e806: 8 total, 8 up, 8 in 2024-03-24T12:13:09.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:09 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:13:10.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:10 smithi119 ceph-mon[33714]: osdmap e807: 8 total, 8 up, 8 in 2024-03-24T12:13:10.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:10 smithi119 ceph-mon[33714]: pgmap v1735: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:10.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:10 smithi114 ceph-mon[30312]: osdmap e807: 8 total, 8 up, 8 in 2024-03-24T12:13:10.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:10 smithi114 ceph-mon[30312]: pgmap v1735: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:10.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:10 smithi114 ceph-mon[42466]: osdmap e807: 8 total, 8 up, 8 in 2024-03-24T12:13:10.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:10 smithi114 ceph-mon[42466]: pgmap v1735: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:11.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:11 smithi114 ceph-mon[30312]: osdmap e808: 8 total, 8 up, 8 in 2024-03-24T12:13:11.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:11 smithi114 ceph-mon[42466]: osdmap e808: 8 total, 8 up, 8 in 2024-03-24T12:13:11.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:11 smithi119 ceph-mon[33714]: osdmap e808: 8 total, 8 up, 8 in 2024-03-24T12:13:12.409 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:12 smithi119 ceph-mon[33714]: osdmap e809: 8 total, 8 up, 8 in 2024-03-24T12:13:12.409 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:12 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:12.409 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:12 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:12.409 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:12 smithi119 ceph-mon[33714]: pgmap v1738: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:12.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[30312]: osdmap e809: 8 total, 8 up, 8 in 2024-03-24T12:13:12.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:12.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[30312]: pgmap v1738: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[42466]: osdmap e809: 8 total, 8 up, 8 in 2024-03-24T12:13:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:12 smithi114 ceph-mon[42466]: pgmap v1738: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:13.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:13 smithi114 ceph-mon[30312]: osdmap e810: 8 total, 8 up, 8 in 2024-03-24T12:13:13.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:13 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:13.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:13 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:13 smithi114 ceph-mon[42466]: osdmap e810: 8 total, 8 up, 8 in 2024-03-24T12:13:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:13 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:13 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:13 smithi119 ceph-mon[33714]: osdmap e810: 8 total, 8 up, 8 in 2024-03-24T12:13:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:13 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:13 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:13.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:13:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:13:14.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[30312]: osdmap e811: 8 total, 8 up, 8 in 2024-03-24T12:13:14.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[30312]: pgmap v1741: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[42466]: osdmap e811: 8 total, 8 up, 8 in 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:13:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-mon[42466]: pgmap v1741: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:14 smithi119 ceph-mon[33714]: osdmap e811: 8 total, 8 up, 8 in 2024-03-24T12:13:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:13:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:13:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:14 smithi119 ceph-mon[33714]: pgmap v1741: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:14.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:13:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:13:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:15.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:15 smithi114 ceph-mon[30312]: osdmap e812: 8 total, 8 up, 8 in 2024-03-24T12:13:15.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:15 smithi114 ceph-mon[42466]: osdmap e812: 8 total, 8 up, 8 in 2024-03-24T12:13:15.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:15 smithi119 ceph-mon[33714]: osdmap e812: 8 total, 8 up, 8 in 2024-03-24T12:13:16.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:16 smithi114 ceph-mon[30312]: pgmap v1743: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:13:16.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:16 smithi114 ceph-mon[42466]: pgmap v1743: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:13:16.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:16 smithi119 ceph-mon[33714]: pgmap v1743: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:13:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:18 smithi119 ceph-mon[33714]: pgmap v1744: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-24T12:13:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:18 smithi119 ceph-mon[33714]: osdmap e813: 8 total, 8 up, 8 in 2024-03-24T12:13:18.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:18 smithi114 ceph-mon[42466]: pgmap v1744: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-24T12:13:18.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:18 smithi114 ceph-mon[42466]: osdmap e813: 8 total, 8 up, 8 in 2024-03-24T12:13:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:18 smithi114 ceph-mon[30312]: pgmap v1744: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-24T12:13:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:18 smithi114 ceph-mon[30312]: osdmap e813: 8 total, 8 up, 8 in 2024-03-24T12:13:20.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:20 smithi119 ceph-mon[33714]: pgmap v1746: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 803 B/s rd, 160 B/s wr, 0 op/s 2024-03-24T12:13:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:20.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:20 smithi114 ceph-mon[42466]: pgmap v1746: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 803 B/s rd, 160 B/s wr, 0 op/s 2024-03-24T12:13:20.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:20 smithi114 ceph-mon[30312]: pgmap v1746: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 803 B/s rd, 160 B/s wr, 0 op/s 2024-03-24T12:13:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:22.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:22 smithi119 ceph-mon[33714]: pgmap v1747: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-24T12:13:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:22 smithi114 ceph-mon[30312]: pgmap v1747: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-24T12:13:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:22 smithi114 ceph-mon[42466]: pgmap v1747: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-24T12:13:23.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:13:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:13:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:24.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:24 smithi119 ceph-mon[33714]: pgmap v1748: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 109 B/s wr, 1 op/s 2024-03-24T12:13:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:24 smithi114 ceph-mon[30312]: pgmap v1748: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 109 B/s wr, 1 op/s 2024-03-24T12:13:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:13:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:13:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:24 smithi114 ceph-mon[42466]: pgmap v1748: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 109 B/s wr, 1 op/s 2024-03-24T12:13:26.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:26 smithi119 ceph-mon[33714]: pgmap v1749: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-24T12:13:26.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:26 smithi114 ceph-mon[30312]: pgmap v1749: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-24T12:13:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:26 smithi114 ceph-mon[42466]: pgmap v1749: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-24T12:13:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:28 smithi119 ceph-mon[33714]: pgmap v1750: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:13:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:28 smithi114 ceph-mon[30312]: pgmap v1750: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:13:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:28 smithi114 ceph-mon[42466]: pgmap v1750: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:13:29.517 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: Running main() from gmock_main.cc 2024-03-24T12:13:29.517 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-24T12:13:29.517 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [----------] Global test environment set-up. 2024-03-24T12:13:29.517 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-24T12:13:29.517 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (1811370 ms) 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5047 ms) 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4037 ms) 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5067 ms) 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7042 ms) 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (1832563 ms total) 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5033 ms) 2024-03-24T12:13:29.518 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6033 ms) 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8079 ms) 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6042 ms) 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4027 ms) 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi114-83801-11. 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: Waiting for snaps to purge. 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19456 ms) 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48670 ms total) 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [----------] Global test environment tear-down 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (1881233 ms total) 2024-03-24T12:13:29.519 INFO:tasks.workunit.client.0.smithi114.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-24T12:13:29.520 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.520 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82830 2024-03-24T12:13:29.520 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82830 2024-03-24T12:13:29.520 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.520 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82340 2024-03-24T12:13:29.520 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82340 2024-03-24T12:13:29.520 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82398 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82398 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82785 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82785 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82439 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82439 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82865 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82865 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=82940 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 82940 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.521 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83418 2024-03-24T12:13:29.522 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83418 2024-03-24T12:13:29.522 INFO:tasks.workunit.client.0.smithi114.stderr:+ for t in "${!pids[@]}" 2024-03-24T12:13:29.522 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=83701 2024-03-24T12:13:29.522 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 83701 2024-03-24T12:13:29.522 INFO:tasks.workunit.client.0.smithi114.stderr:+ exit 0 2024-03-24T12:13:29.522 INFO:tasks.workunit.client.0.smithi114.stderr:+ cleanup 2024-03-24T12:13:29.522 INFO:tasks.workunit.client.0.smithi114.stderr:+ pkill -P 82334 2024-03-24T12:13:29.561 INFO:tasks.workunit.client.0.smithi114.stderr:+ true 2024-03-24T12:13:29.562 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-24T12:13:29.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-24T12:13:29.595 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-24T12:13:29.596 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-24T12:13:29.596 DEBUG:teuthology.orchestra.run.smithi114:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f73f1dfd96849300c4322bd7259a3ab37f94c200 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-24T12:13:29.655 INFO:tasks.workunit.client.0.smithi114.stderr:++ uuidgen 2024-03-24T12:13:29.669 INFO:tasks.workunit.client.0.smithi114.stderr:+ p=5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:13:29.669 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool create 5e219aa2-0962-4332-a6f6-f7f15fae1d00 12 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.751+0000 7f86af5bc640 1 -- 172.21.15.114:0/4013850776 >> v1:172.21.15.114:6790/0 conn(0x7f86a8109e90 legacy=0x7f86a810c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.751+0000 7f86af5bc640 1 -- 172.21.15.114:0/4013850776 shutdown_connections 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.751+0000 7f86af5bc640 1 -- 172.21.15.114:0/4013850776 >> 172.21.15.114:0/4013850776 conn(0x7f86a80751e0 msgr2=0x7f86a80755f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.751+0000 7f86af5bc640 1 -- 172.21.15.114:0/4013850776 shutdown_connections 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.751+0000 7f86af5bc640 1 -- 172.21.15.114:0/4013850776 wait complete. 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f86af5bc640 1 Processor -- start 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f86af5bc640 1 -- start start 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f86af5bc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86a81a8100 con 0x7f86a8109e90 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f86af5bc640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86a81a9300 con 0x7f86a8105fd0 2024-03-24T12:13:29.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f86af5bc640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86a81aa500 con 0x7f86a810da60 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f86acb57640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f86a8109e90 0x7f86a81a3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37862/0 (socket says 172.21.15.114:37862) 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f86acb57640 1 -- 172.21.15.114:0/1616151079 learned_addr learned my addr 172.21.15.114:0/1616151079 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503484875 0 0) 0x7f86a81a8100 con 0x7f86a8109e90 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8694003160 con 0x7f86a8109e90 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585997690 0 0) 0x7f86a81aa500 con 0x7f86a810da60 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a81a8100 con 0x7f86a810da60 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622306132 0 0) 0x7f86a81a9300 con 0x7f86a8105fd0 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.752+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a81aa500 con 0x7f86a8105fd0 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 692419804 0 0) 0x7f8694003160 con 0x7f86a8109e90 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86a81a9300 con 0x7f86a8109e90 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f86980015d0 con 0x7f86a8109e90 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3666738756 0 0) 0x7f86a81a8100 con 0x7f86a810da60 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8694003160 con 0x7f86a810da60 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f86a0001770 con 0x7f86a810da60 2024-03-24T12:13:29.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1901680351 0 0) 0x7f86a81a9300 con 0x7f86a8109e90 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 >> v1:172.21.15.114:6790/0 conn(0x7f86a810da60 legacy=0x7f86a81a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 >> v1:172.21.15.119:6789/0 conn(0x7f86a8105fd0 legacy=0x7f86a80fff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a81ab700 con 0x7f86a8109e90 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a81aa6f0 con 0x7f86a8109e90 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a81aab80 con 0x7f86a8109e90 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f8698001a70 con 0x7f86a8109e90 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.753+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f8698004e70 con 0x7f86a8109e90 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.754+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c005080 con 0x7f86a8109e90 2024-03-24T12:13:29.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.754+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f8698004410 con 0x7f86a8109e90 2024-03-24T12:13:29.757 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.755+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(814..814 src has 254..814) v4 ==== 7414+0+0 (unknown 2393148755 0 0) 0x7f86980889e0 con 0x7f86a8109e90 2024-03-24T12:13:29.759 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.759+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f8698055a40 con 0x7f86a8109e90 2024-03-24T12:13:29.938 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:29.935+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12} v 0) v1 -- 0x7f868c0058b0 con 0x7f86a8109e90 2024-03-24T12:13:30.516 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.516+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]=0 pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' created v815) v1 ==== 176+0+0 (unknown 3752702961 0 0) 0x7f8698053db0 con 0x7f86a8109e90 2024-03-24T12:13:30.630 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.629+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12} v 0) v1 -- 0x7f868c0053e0 con 0x7f86a8109e90 2024-03-24T12:13:30.630 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.630+0000 7f869e7fc640 1 -- 172.21.15.114:0/1616151079 <== mon.0 v1:172.21.15.114:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]=0 pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' already exists v815) v1 ==== 183+0+0 (unknown 1957135605 0 0) 0x7f868c0053e0 con 0x7f86a8109e90 2024-03-24T12:13:30.630 INFO:tasks.workunit.client.0.smithi114.stderr:pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' already exists 2024-03-24T12:13:30.632 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.632+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 >> v1:172.21.15.114:6800/1253397276 conn(0x7f869406e410 legacy=0x7f86940708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:30.632 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.632+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 >> v1:172.21.15.114:6789/0 conn(0x7f86a8109e90 legacy=0x7f86a81a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:30.632 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.632+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 shutdown_connections 2024-03-24T12:13:30.632 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.632+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 >> 172.21.15.114:0/1616151079 conn(0x7f86a80751e0 msgr2=0x7f86a810def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:13:30.632 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.632+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 shutdown_connections 2024-03-24T12:13:30.632 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.632+0000 7f86af5bc640 1 -- 172.21.15.114:0/1616151079 wait complete. 2024-03-24T12:13:30.641 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 5e219aa2-0962-4332-a6f6-f7f15fae1d00 max_objects 10 2024-03-24T12:13:30.722 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.721+0000 7f2230c84640 1 -- 172.21.15.114:0/1212291436 >> v1:172.21.15.114:6789/0 conn(0x7f222c1021a0 legacy=0x7f222c102580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:30.722 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.721+0000 7f2230c84640 1 -- 172.21.15.114:0/1212291436 shutdown_connections 2024-03-24T12:13:30.722 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.721+0000 7f2230c84640 1 -- 172.21.15.114:0/1212291436 >> 172.21.15.114:0/1212291436 conn(0x7f222c0fb980 msgr2=0x7f222c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:13:30.722 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.721+0000 7f2230c84640 1 -- 172.21.15.114:0/1212291436 shutdown_connections 2024-03-24T12:13:30.722 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.721+0000 7f2230c84640 1 -- 172.21.15.114:0/1212291436 wait complete. 2024-03-24T12:13:30.722 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f2230c84640 1 Processor -- start 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f2230c84640 1 -- start start 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f2230c84640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f222c1a5e00 con 0x7f222c1021a0 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f2230c84640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f222c1a7000 con 0x7f222c10dab0 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f2230c84640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f222c1a8200 con 0x7f222c109fa0 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f2221d9b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f222c109fa0 0x7f222c1a0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35020/0 (socket says 172.21.15.114:35020) 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f222a59c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f222c1021a0 0x7f222c109690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58118/0 (socket says 172.21.15.114:58118) 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.722+0000 7f2221d9b640 1 -- 172.21.15.114:0/1498621124 learned_addr learned my addr 172.21.15.114:0/1498621124 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354067153 0 0) 0x7f222c1a5e00 con 0x7f222c1021a0 2024-03-24T12:13:30.725 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2218003240 con 0x7f222c1021a0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007242859 0 0) 0x7f222c1a8200 con 0x7f222c109fa0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c1a5e00 con 0x7f222c109fa0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317251851 0 0) 0x7f222c1a7000 con 0x7f222c10dab0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c1a8200 con 0x7f222c10dab0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992248982 0 0) 0x7f2218003240 con 0x7f222c1021a0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c1a7000 con 0x7f222c1021a0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f221c004380 con 0x7f222c1021a0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2389546328 0 0) 0x7f222c1a5e00 con 0x7f222c109fa0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2218003240 con 0x7f222c109fa0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f22140017f0 con 0x7f222c109fa0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3173392888 0 0) 0x7f222c1a8200 con 0x7f222c10dab0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c1a5e00 con 0x7f222c10dab0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2224001d70 con 0x7f222c10dab0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3031185565 0 0) 0x7f222c1a7000 con 0x7f222c1021a0 2024-03-24T12:13:30.726 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 >> v1:172.21.15.114:6790/0 conn(0x7f222c109fa0 legacy=0x7f222c1a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 >> v1:172.21.15.119:6789/0 conn(0x7f222c10dab0 legacy=0x7f222c1a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c1a9400 con 0x7f222c1021a0 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c1a83f0 con 0x7f222c1021a0 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c1a8880 con 0x7f222c1021a0 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f221c001a50 con 0x7f222c1021a0 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.723+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f221c004d10 con 0x7f222c1021a0 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.724+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c005070 con 0x7f222c1021a0 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.725+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f221c0035f0 con 0x7f222c1021a0 2024-03-24T12:13:30.727 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.725+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(815..815 src has 254..815) v4 ==== 7789+0+0 (unknown 2214923462 0 0) 0x7f221c088a00 con 0x7f222c1021a0 2024-03-24T12:13:30.729 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.730+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f221c055950 con 0x7f222c1021a0 2024-03-24T12:13:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:30 smithi119 ceph-mon[33714]: pgmap v1751: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-24T12:13:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:30 smithi119 ceph-mon[33714]: osdmap e814: 8 total, 8 up, 8 in 2024-03-24T12:13:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:30 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]: dispatch 2024-03-24T12:13:30.903 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:30.903+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f220c005320 con 0x7f222c1021a0 2024-03-24T12:13:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:30 smithi114 ceph-mon[30312]: pgmap v1751: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-24T12:13:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:30 smithi114 ceph-mon[30312]: osdmap e814: 8 total, 8 up, 8 in 2024-03-24T12:13:30.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:30 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]: dispatch 2024-03-24T12:13:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:30 smithi114 ceph-mon[42466]: pgmap v1751: 196 pgs: 196 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-24T12:13:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:30 smithi114 ceph-mon[42466]: osdmap e814: 8 total, 8 up, 8 in 2024-03-24T12:13:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:30 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]: dispatch 2024-03-24T12:13:31.521 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:31.521+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v816) v1 ==== 223+0+0 (unknown 2738590256 0 0) 0x7f221c001150 con 0x7f222c1021a0 2024-03-24T12:13:31.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:31.636+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f220c001090 con 0x7f222c1021a0 2024-03-24T12:13:31.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]': finished 2024-03-24T12:13:31.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:31 smithi119 ceph-mon[33714]: osdmap e815: 8 total, 8 up, 8 in 2024-03-24T12:13:31.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]: dispatch 2024-03-24T12:13:31.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:31 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:13:31.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]': finished 2024-03-24T12:13:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[42466]: osdmap e815: 8 total, 8 up, 8 in 2024-03-24T12:13:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]: dispatch 2024-03-24T12:13:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:13:31.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]': finished 2024-03-24T12:13:31.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[30312]: osdmap e815: 8 total, 8 up, 8 in 2024-03-24T12:13:31.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1616151079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pg_num": 12}]: dispatch 2024-03-24T12:13:31.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:31 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:13:32.532 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.531+0000 7f2223fff640 1 -- 172.21.15.114:0/1498621124 <== mon.0 v1:172.21.15.114:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v817) v1 ==== 223+0+0 (unknown 3934140224 0 0) 0x7f221c053cc0 con 0x7f222c1021a0 2024-03-24T12:13:32.532 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_objects = 10 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:13:32.534 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.534+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 >> v1:172.21.15.114:6800/1253397276 conn(0x7f221806e6e0 legacy=0x7f2218070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:32.534 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.534+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 >> v1:172.21.15.114:6789/0 conn(0x7f222c1021a0 legacy=0x7f222c109690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:32.534 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.534+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 shutdown_connections 2024-03-24T12:13:32.534 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.534+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 >> 172.21.15.114:0/1498621124 conn(0x7f222c0fb980 msgr2=0x7f222c10ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:13:32.535 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.535+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 shutdown_connections 2024-03-24T12:13:32.536 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.535+0000 7f2230c84640 1 -- 172.21.15.114:0/1498621124 wait complete. 2024-03-24T12:13:32.547 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool application enable 5e219aa2-0962-4332-a6f6-f7f15fae1d00 rados 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.633+0000 7ffb0700f640 1 -- 172.21.15.114:0/4274504953 >> v1:172.21.15.114:6789/0 conn(0x7ffb00101230 legacy=0x7ffb00101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.633+0000 7ffb0700f640 1 -- 172.21.15.114:0/4274504953 shutdown_connections 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.633+0000 7ffb0700f640 1 -- 172.21.15.114:0/4274504953 >> 172.21.15.114:0/4274504953 conn(0x7ffb000fca70 msgr2=0x7ffb000fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.633+0000 7ffb0700f640 1 -- 172.21.15.114:0/4274504953 shutdown_connections 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.633+0000 7ffb0700f640 1 -- 172.21.15.114:0/4274504953 wait complete. 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffb0700f640 1 Processor -- start 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffb0700f640 1 -- start start 2024-03-24T12:13:32.636 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffb0700f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffb001a8090 con 0x7ffb00109eb0 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffb0700f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffb001a9290 con 0x7ffb00101230 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffb0700f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffb001aa490 con 0x7ffb0010da20 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffaf7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffb00109eb0 0x7ffb001a3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58144/0 (socket says 172.21.15.114:58144) 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffaf7fff640 1 -- 172.21.15.114:0/4180276874 learned_addr learned my addr 172.21.15.114:0/4180276874 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffb055ac640 1 --1- 172.21.15.114:0/4180276874 >> v1:172.21.15.114:6790/0 conn(0x7ffb0010da20 0x7ffb001a68b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35032/0 (socket says 172.21.15.114:35032) 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799073026 0 0) 0x7ffb001a8090 con 0x7ffb00109eb0 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae8003520 con 0x7ffb00109eb0 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196940712 0 0) 0x7ffb001aa490 con 0x7ffb0010da20 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.634+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb001a8090 con 0x7ffb0010da20 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802298814 0 0) 0x7ffb001a9290 con 0x7ffb00101230 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb001aa490 con 0x7ffb00101230 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3500091495 0 0) 0x7ffae8003520 con 0x7ffb00109eb0 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb001a9290 con 0x7ffb00109eb0 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616977603 0 0) 0x7ffb001a8090 con 0x7ffb0010da20 2024-03-24T12:13:32.637 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae8003520 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ffaf00019d0 con 0x7ffb00109eb0 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ffaf8001bb0 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3717715650 0 0) 0x7ffae8003520 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 >> v1:172.21.15.119:6789/0 conn(0x7ffb00101230 legacy=0x7ffb00106220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 >> v1:172.21.15.114:6789/0 conn(0x7ffb00109eb0 legacy=0x7ffb001a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb001ab690 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb001aa680 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7ffaf80020e0 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7ffaf8005ce0 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.635+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb001aab10 con 0x7ffb0010da20 2024-03-24T12:13:32.638 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.636+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb00102700 con 0x7ffb0010da20 2024-03-24T12:13:32.639 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.636+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7ffaf8003970 con 0x7ffb0010da20 2024-03-24T12:13:32.639 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.637+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(817..817 src has 254..817) v4 ==== 7789+0+0 (unknown 3546662058 0 0) 0x7ffaf8088a00 con 0x7ffb0010da20 2024-03-24T12:13:32.641 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.642+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7ffaf8055950 con 0x7ffb0010da20 2024-03-24T12:13:32.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:32 smithi119 ceph-mon[33714]: pgmap v1754: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:13:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:32 smithi119 ceph-mon[33714]: osdmap e816: 8 total, 8 up, 8 in 2024-03-24T12:13:32.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:32 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:13:32.816 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:32.816+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"} v 0) v1 -- 0x7ffb00068260 con 0x7ffb0010da20 2024-03-24T12:13:32.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[42466]: pgmap v1754: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:32.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:13:32.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[42466]: osdmap e816: 8 total, 8 up, 8 in 2024-03-24T12:13:32.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:13:32.889 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[30312]: pgmap v1754: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:32.889 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:13:32.890 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[30312]: osdmap e816: 8 total, 8 up, 8 in 2024-03-24T12:13:32.890 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:32 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:13:33.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:13:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:13:33.546 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:33.546+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]=0 enabled application 'rados' on pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' v818) v1 ==== 213+0+0 (unknown 851151163 0 0) 0x7ffaf8053cc0 con 0x7ffb0010da20 2024-03-24T12:13:33.652 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:33.652+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"} v 0) v1 -- 0x7ffb001aadc0 con 0x7ffb0010da20 2024-03-24T12:13:33.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:13:33.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:33 smithi119 ceph-mon[33714]: osdmap e817: 8 total, 8 up, 8 in 2024-03-24T12:13:33.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:33 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4180276874' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:33.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:33 smithi119 ceph-mon[33714]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:33.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:13:33.900 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[30312]: osdmap e817: 8 total, 8 up, 8 in 2024-03-24T12:13:33.901 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4180276874' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:33.901 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[30312]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:33.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1498621124' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:13:33.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[42466]: osdmap e817: 8 total, 8 up, 8 in 2024-03-24T12:13:33.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4180276874' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:33.901 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:33 smithi114 ceph-mon[42466]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:34.548 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:34.547+0000 7ffaf5ffb640 1 -- 172.21.15.114:0/4180276874 <== mon.2 v1:172.21.15.114:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]=0 enabled application 'rados' on pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' v819) v1 ==== 213+0+0 (unknown 144343005 0 0) 0x7ffaf80554e0 con 0x7ffb0010da20 2024-03-24T12:13:34.548 INFO:tasks.workunit.client.0.smithi114.stderr:enabled application 'rados' on pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' 2024-03-24T12:13:34.550 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:34.550+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 >> v1:172.21.15.114:6800/1253397276 conn(0x7ffae806e270 legacy=0x7ffae8070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:34.551 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:34.550+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 >> v1:172.21.15.114:6790/0 conn(0x7ffb0010da20 legacy=0x7ffb001a68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:13:34.551 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:34.550+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 shutdown_connections 2024-03-24T12:13:34.551 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:34.550+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 >> 172.21.15.114:0/4180276874 conn(0x7ffb000fca70 msgr2=0x7ffb00107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:13:34.551 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:34.550+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 shutdown_connections 2024-03-24T12:13:34.551 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:13:34.550+0000 7ffb0700f640 1 -- 172.21.15.114:0/4180276874 wait complete. 2024-03-24T12:13:34.563 INFO:tasks.workunit.client.0.smithi114.stderr:++ seq 1 10 2024-03-24T12:13:34.579 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.579 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj1 /etc/passwd 2024-03-24T12:13:34.614 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.614 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj2 /etc/passwd 2024-03-24T12:13:34.648 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.648 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj3 /etc/passwd 2024-03-24T12:13:34.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:34.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[42466]: pgmap v1757: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:34.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[42466]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]': finished 2024-03-24T12:13:34.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[42466]: osdmap e818: 8 total, 8 up, 8 in 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/4180276874' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[42466]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[30312]: pgmap v1757: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[30312]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]': finished 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[30312]: osdmap e818: 8 total, 8 up, 8 in 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/4180276874' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:34.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-mon[30312]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:34.682 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.682 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj4 /etc/passwd 2024-03-24T12:13:34.714 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.715 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj5 /etc/passwd 2024-03-24T12:13:34.748 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.748 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj6 /etc/passwd 2024-03-24T12:13:34.781 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.781 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj7 /etc/passwd 2024-03-24T12:13:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:34 smithi119 ceph-mon[33714]: pgmap v1757: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:34 smithi119 ceph-mon[33714]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]': finished 2024-03-24T12:13:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:34 smithi119 ceph-mon[33714]: osdmap e818: 8 total, 8 up, 8 in 2024-03-24T12:13:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:34 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/4180276874' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:34 smithi119 ceph-mon[33714]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]: dispatch 2024-03-24T12:13:34.813 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.814 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj8 /etc/passwd 2024-03-24T12:13:34.846 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.846 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj9 /etc/passwd 2024-03-24T12:13:34.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:13:34.880 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put obj10 /etc/passwd 2024-03-24T12:13:34.914 INFO:tasks.workunit.client.0.smithi114.stderr:+ sleep 30 2024-03-24T12:13:34.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:13:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:13:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:35.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:35 smithi114 ceph-mon[42466]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]': finished 2024-03-24T12:13:35.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:35 smithi114 ceph-mon[42466]: osdmap e819: 8 total, 8 up, 8 in 2024-03-24T12:13:35.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:35.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:35 smithi114 ceph-mon[30312]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]': finished 2024-03-24T12:13:35.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:35 smithi114 ceph-mon[30312]: osdmap e819: 8 total, 8 up, 8 in 2024-03-24T12:13:35.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:36.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:35 smithi119 ceph-mon[33714]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "app": "rados"}]': finished 2024-03-24T12:13:36.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:35 smithi119 ceph-mon[33714]: osdmap e819: 8 total, 8 up, 8 in 2024-03-24T12:13:36.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:36.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:36 smithi114 ceph-mon[42466]: pgmap v1760: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:13:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:36 smithi114 ceph-mon[42466]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:13:36.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:36 smithi114 ceph-mon[30312]: pgmap v1760: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:13:36.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:36 smithi114 ceph-mon[30312]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:13:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:36 smithi119 ceph-mon[33714]: pgmap v1760: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 908 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:13:37.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:36 smithi119 ceph-mon[33714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:13:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[30312]: pgmap v1761: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 864 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-24T12:13:38.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[30312]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' is full (reached quota's max_objects: 10) 2024-03-24T12:13:38.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[30312]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:13:38.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[30312]: osdmap e820: 8 total, 8 up, 8 in 2024-03-24T12:13:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[42466]: pgmap v1761: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 864 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-24T12:13:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[42466]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' is full (reached quota's max_objects: 10) 2024-03-24T12:13:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[42466]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:13:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:38 smithi114 ceph-mon[42466]: osdmap e820: 8 total, 8 up, 8 in 2024-03-24T12:13:39.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:38 smithi119 ceph-mon[33714]: pgmap v1761: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 864 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-24T12:13:39.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:38 smithi119 ceph-mon[33714]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' is full (reached quota's max_objects: 10) 2024-03-24T12:13:39.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:38 smithi119 ceph-mon[33714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:13:39.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:38 smithi119 ceph-mon[33714]: osdmap e820: 8 total, 8 up, 8 in 2024-03-24T12:13:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:40 smithi114 ceph-mon[30312]: pgmap v1763: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-24T12:13:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:40 smithi114 ceph-mon[42466]: pgmap v1763: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-24T12:13:41.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:40 smithi119 ceph-mon[33714]: pgmap v1763: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-24T12:13:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:42 smithi114 ceph-mon[30312]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-24T12:13:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:42 smithi114 ceph-mon[42466]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-24T12:13:43.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:42 smithi119 ceph-mon[33714]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-24T12:13:43.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:13:43 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:13:44.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:44.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:44 smithi114 ceph-mon[42466]: pgmap v1765: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-24T12:13:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:44 smithi114 ceph-mon[30312]: pgmap v1765: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-24T12:13:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:13:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:13:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:45.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:44 smithi119 ceph-mon[33714]: pgmap v1765: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-24T12:13:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:46 smithi114 ceph-mon[30312]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:13:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:46 smithi114 ceph-mon[42466]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:13:47.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:46 smithi119 ceph-mon[33714]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:13:48.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:48 smithi114 ceph-mon[30312]: pgmap v1767: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:13:48.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:48 smithi114 ceph-mon[42466]: pgmap v1767: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:13:49.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:48 smithi119 ceph-mon[33714]: pgmap v1767: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:13:50.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:50 smithi114 ceph-mon[30312]: pgmap v1768: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-24T12:13:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:50 smithi114 ceph-mon[42466]: pgmap v1768: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-24T12:13:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:50 smithi119 ceph-mon[33714]: pgmap v1768: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-24T12:13:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:13:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:13:52.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:52 smithi114 ceph-mon[30312]: pgmap v1769: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:52 smithi114 ceph-mon[42466]: pgmap v1769: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:53.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:52 smithi119 ceph-mon[33714]: pgmap v1769: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:53.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:13:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:13:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:54 smithi114 ceph-mon[30312]: pgmap v1770: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:13:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:13:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:13:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:13:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:54 smithi114 ceph-mon[42466]: pgmap v1770: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:13:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:13:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:54 smithi119 ceph-mon[33714]: pgmap v1770: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:13:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:56 smithi114 ceph-mon[30312]: pgmap v1771: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:13:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:56 smithi114 ceph-mon[42466]: pgmap v1771: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:13:57.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:56 smithi119 ceph-mon[33714]: pgmap v1771: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:13:58.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:13:58 smithi114 ceph-mon[30312]: pgmap v1772: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:13:58 smithi114 ceph-mon[42466]: pgmap v1772: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:13:59.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:13:58 smithi119 ceph-mon[33714]: pgmap v1772: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:00.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:00 smithi114 ceph-mon[30312]: pgmap v1773: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:00 smithi114 ceph-mon[42466]: pgmap v1773: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:01.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:00 smithi119 ceph-mon[33714]: pgmap v1773: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:02.855 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:02 smithi119 ceph-mon[33714]: pgmap v1774: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:02.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:02 smithi114 ceph-mon[30312]: pgmap v1774: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:02.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:02 smithi114 ceph-mon[42466]: pgmap v1774: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:03.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:14:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:14:04.915 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=159509 2024-03-24T12:14:04.915 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 5e219aa2-0962-4332-a6f6-f7f15fae1d00 max_objects 100 2024-03-24T12:14:04.915 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put onemore /etc/passwd 2024-03-24T12:14:04.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:04.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:04 smithi114 ceph-mon[42466]: pgmap v1775: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:04 smithi114 ceph-mon[30312]: pgmap v1775: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:14:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:14:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:05.009 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.009+0000 7f402a310640 1 -- 172.21.15.114:0/379097847 >> v1:172.21.15.114:6789/0 conn(0x7f40241021c0 legacy=0x7f40241025a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:05.009 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.010+0000 7f402a310640 1 -- 172.21.15.114:0/379097847 shutdown_connections 2024-03-24T12:14:05.009 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.010+0000 7f402a310640 1 -- 172.21.15.114:0/379097847 >> 172.21.15.114:0/379097847 conn(0x7f40240fb9e0 msgr2=0x7f40240fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:05.009 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.010+0000 7f402a310640 1 -- 172.21.15.114:0/379097847 shutdown_connections 2024-03-24T12:14:05.009 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.010+0000 7f402a310640 1 -- 172.21.15.114:0/379097847 wait complete. 2024-03-24T12:14:05.009 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.010+0000 7f402a310640 1 Processor -- start 2024-03-24T12:14:05.009 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.010+0000 7f402a310640 1 -- start start 2024-03-24T12:14:05.012 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f402a310640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40241a5e20 con 0x7f40241021c0 2024-03-24T12:14:05.012 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f402a310640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40241a7020 con 0x7f402410dad0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f402a310640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40241a8220 con 0x7f4024109fc0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f4023fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f40241021c0 0x7f40241096b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57984/0 (socket says 172.21.15.114:57984) 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f4023fff640 1 -- 172.21.15.114:0/2775078193 learned_addr learned my addr 172.21.15.114:0/2775078193 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129878965 0 0) 0x7f40241a5e20 con 0x7f40241021c0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c001fe0 con 0x7f40241021c0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347407363 0 0) 0x7f40241a8220 con 0x7f4024109fc0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40241a5e20 con 0x7f4024109fc0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808913846 0 0) 0x7f40241a7020 con 0x7f402410dad0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40241a8220 con 0x7f402410dad0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3865117946 0 0) 0x7f40241a5e20 con 0x7f4024109fc0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40241a7020 con 0x7f4024109fc0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120871829 0 0) 0x7f400c001fe0 con 0x7f40241021c0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40241a5e20 con 0x7f40241021c0 2024-03-24T12:14:05.013 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4018001830 con 0x7f4024109fc0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f40140015d0 con 0x7f40241021c0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 55205475 0 0) 0x7f40241a8220 con 0x7f402410dad0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f400c001fe0 con 0x7f402410dad0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f401c0013f0 con 0x7f402410dad0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3681720712 0 0) 0x7f40241a7020 con 0x7f4024109fc0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 >> v1:172.21.15.119:6789/0 conn(0x7f402410dad0 legacy=0x7f40241a4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 >> v1:172.21.15.114:6789/0 conn(0x7f40241021c0 legacy=0x7f40241096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40241a9420 con 0x7f4024109fc0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.011+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40241a8410 con 0x7f4024109fc0 2024-03-24T12:14:05.014 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.012+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40241a88a0 con 0x7f4024109fc0 2024-03-24T12:14:05.015 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.012+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f4018003a80 con 0x7f4024109fc0 2024-03-24T12:14:05.015 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.012+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4018004960 con 0x7f4024109fc0 2024-03-24T12:14:05.015 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.012+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4010005070 con 0x7f4024109fc0 2024-03-24T12:14:05.015 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.013+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f40180035f0 con 0x7f4024109fc0 2024-03-24T12:14:05.015 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.013+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(820..820 src has 254..820) v4 ==== 7802+0+0 (unknown 2523911240 0 0) 0x7f40180876b0 con 0x7f4024109fc0 2024-03-24T12:14:05.015 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.014+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=821}) v3 -- 0x7f40241a7020 con 0x7f4024109fc0 2024-03-24T12:14:05.017 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.018+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f4018054590 con 0x7f4024109fc0 2024-03-24T12:14:05.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:04 smithi119 ceph-mon[33714]: pgmap v1775: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:05.191 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.191+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f4010005320 con 0x7f4024109fc0 2024-03-24T12:14:05.607 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.608+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 10 ==== osd_map(821..821 src has 254..821) v4 ==== 628+0+0 (unknown 4066850247 0 0) 0x7f4018054ec0 con 0x7f4024109fc0 2024-03-24T12:14:05.607 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.608+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=822}) v3 -- 0x7f40241a5e20 con 0x7f4024109fc0 2024-03-24T12:14:05.613 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.614+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v821) v1 ==== 225+0+0 (unknown 179840161 0 0) 0x7f4018052900 con 0x7f4024109fc0 2024-03-24T12:14:05.718 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:05.719+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f4010001090 con 0x7f4024109fc0 2024-03-24T12:14:05.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:05 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2775078193' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:05.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:05 smithi114 ceph-mon[30312]: from='client.56354 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:05.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:05 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2775078193' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:05 smithi114 ceph-mon[42466]: from='client.56354 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:06.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:05 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2775078193' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:06.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:05 smithi119 ceph-mon[33714]: from='client.56354 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:06.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:06.607 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.608+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 12 ==== osd_map(822..822 src has 254..822) v4 ==== 628+0+0 (unknown 2393377304 0 0) 0x7f401805cf60 con 0x7f4024109fc0 2024-03-24T12:14:06.607 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.608+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=823}) v3 -- 0x7f40241a8220 con 0x7f4024109fc0 2024-03-24T12:14:06.611 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.612+0000 7f40217fa640 1 -- 172.21.15.114:0/2775078193 <== mon.2 v1:172.21.15.114:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v822) v1 ==== 225+0+0 (unknown 1939693609 0 0) 0x7f4018054120 con 0x7f4024109fc0 2024-03-24T12:14:06.611 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_objects = 100 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:14:06.614 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.615+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 >> v1:172.21.15.114:6800/1253397276 conn(0x7f400c06e690 legacy=0x7f400c070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:06.614 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.615+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 >> v1:172.21.15.114:6790/0 conn(0x7f4024109fc0 legacy=0x7f40241a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:06.615 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.616+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 shutdown_connections 2024-03-24T12:14:06.615 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.616+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 >> 172.21.15.114:0/2775078193 conn(0x7f40240fb9e0 msgr2=0x7f402410ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:06.615 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.616+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 shutdown_connections 2024-03-24T12:14:06.615 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:06.616+0000 7f402a310640 1 -- 172.21.15.114:0/2775078193 wait complete. 2024-03-24T12:14:06.626 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 159509 2024-03-24T12:14:06.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[42466]: pgmap v1776: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:06.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[42466]: from='client.56354 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]': finished 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[42466]: osdmap e821: 8 total, 8 up, 8 in 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2775078193' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[42466]: from='client.56354 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[30312]: pgmap v1776: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[30312]: from='client.56354 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]': finished 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[30312]: osdmap e821: 8 total, 8 up, 8 in 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2775078193' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:06.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:06 smithi114 ceph-mon[30312]: from='client.56354 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:06 smithi119 ceph-mon[33714]: pgmap v1776: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:06 smithi119 ceph-mon[33714]: from='client.56354 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]': finished 2024-03-24T12:14:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:06 smithi119 ceph-mon[33714]: osdmap e821: 8 total, 8 up, 8 in 2024-03-24T12:14:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:06 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2775078193' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:06 smithi119 ceph-mon[33714]: from='client.56354 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]: dispatch 2024-03-24T12:14:07.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:07 smithi114 ceph-mon[30312]: from='client.56354 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]': finished 2024-03-24T12:14:07.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:07 smithi114 ceph-mon[30312]: osdmap e822: 8 total, 8 up, 8 in 2024-03-24T12:14:07.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:07 smithi114 ceph-mon[42466]: from='client.56354 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]': finished 2024-03-24T12:14:07.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:07 smithi114 ceph-mon[42466]: osdmap e822: 8 total, 8 up, 8 in 2024-03-24T12:14:08.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:07 smithi119 ceph-mon[33714]: from='client.56354 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "100"}]': finished 2024-03-24T12:14:08.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:07 smithi119 ceph-mon[33714]: osdmap e822: 8 total, 8 up, 8 in 2024-03-24T12:14:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 0 -ne 0 ']' 2024-03-24T12:14:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+ true 2024-03-24T12:14:08.228 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put twomore /etc/passwd 2024-03-24T12:14:08.263 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 5e219aa2-0962-4332-a6f6-f7f15fae1d00 max_bytes 100 2024-03-24T12:14:08.348 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.348+0000 7f4ae5165640 1 -- 172.21.15.114:0/4010892303 >> v1:172.21.15.114:6789/0 conn(0x7f4ae0100140 legacy=0x7f4ae0100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:08.348 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.348+0000 7f4ae5165640 1 -- 172.21.15.114:0/4010892303 shutdown_connections 2024-03-24T12:14:08.348 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.348+0000 7f4ae5165640 1 -- 172.21.15.114:0/4010892303 >> 172.21.15.114:0/4010892303 conn(0x7f4ae00fb980 msgr2=0x7f4ae00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:08.348 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.348+0000 7f4ae5165640 1 -- 172.21.15.114:0/4010892303 shutdown_connections 2024-03-24T12:14:08.348 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.348+0000 7f4ae5165640 1 -- 172.21.15.114:0/4010892303 wait complete. 2024-03-24T12:14:08.348 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ae5165640 1 Processor -- start 2024-03-24T12:14:08.348 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ae5165640 1 -- start start 2024-03-24T12:14:08.352 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ae5165640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ae01a4da0 con 0x7f4ae010c930 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ae5165640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ae01a5fa0 con 0x7f4ae0100140 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ae5165640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4ae01a71a0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ade59c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4ae0108dc0 0x7f4ae019fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33104/0 (socket says 172.21.15.114:33104) 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4adf59e640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4ae010c930 0x7f4ae01a34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58026/0 (socket says 172.21.15.114:58026) 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ade59c640 1 -- 172.21.15.114:0/1355229407 learned_addr learned my addr 172.21.15.114:0/1355229407 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956082293 0 0) 0x7f4ae01a71a0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac8003520 con 0x7f4ae0108dc0 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122733101 0 0) 0x7f4ae01a4da0 con 0x7f4ae010c930 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae01a71a0 con 0x7f4ae010c930 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777184901 0 0) 0x7f4ae01a5fa0 con 0x7f4ae0100140 2024-03-24T12:14:08.353 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.349+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae01a4da0 con 0x7f4ae0100140 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1393517217 0 0) 0x7f4ac8003520 con 0x7f4ae0108dc0 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ae01a5fa0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4ad0001930 con 0x7f4ae0108dc0 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3373502150 0 0) 0x7f4ae01a71a0 con 0x7f4ae010c930 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac8003520 con 0x7f4ae010c930 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4ad8001bd0 con 0x7f4ae010c930 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310829061 0 0) 0x7f4ae01a4da0 con 0x7f4ae0100140 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ae01a71a0 con 0x7f4ae0100140 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4ad4004480 con 0x7f4ae0100140 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3651821159 0 0) 0x7f4ae01a5fa0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 >> v1:172.21.15.119:6789/0 conn(0x7f4ae0100140 legacy=0x7f4ae01053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:08.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 >> v1:172.21.15.114:6789/0 conn(0x7f4ae010c930 legacy=0x7f4ae01a34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:08.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae01a83a0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae01a4f90 con 0x7f4ae0108dc0 2024-03-24T12:14:08.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae01a5420 con 0x7f4ae0108dc0 2024-03-24T12:14:08.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f4ad0001cc0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.350+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f4ad0005c10 con 0x7f4ae0108dc0 2024-03-24T12:14:08.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.351+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc005080 con 0x7f4ae0108dc0 2024-03-24T12:14:08.356 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.351+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f4ad00038c0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.356 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.352+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(823..823 src has 254..823) v4 ==== 7802+0+0 (unknown 2826636148 0 0) 0x7f4ad00889c0 con 0x7f4ae0108dc0 2024-03-24T12:14:08.356 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.356+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f4ad0055650 con 0x7f4ae0108dc0 2024-03-24T12:14:08.533 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:08.533+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f4acc005330 con 0x7f4ae0108dc0 2024-03-24T12:14:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[30312]: pgmap v1779: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[30312]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' no longer out of quota; removing NO_QUOTA flag 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[30312]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[30312]: osdmap e823: 8 total, 8 up, 8 in 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1355229407' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[30312]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[42466]: pgmap v1779: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[42466]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' no longer out of quota; removing NO_QUOTA flag 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[42466]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[42466]: osdmap e823: 8 total, 8 up, 8 in 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1355229407' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:08 smithi114 ceph-mon[42466]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:08 smithi119 ceph-mon[33714]: pgmap v1779: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:08 smithi119 ceph-mon[33714]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' no longer out of quota; removing NO_QUOTA flag 2024-03-24T12:14:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:08 smithi119 ceph-mon[33714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:14:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:08 smithi119 ceph-mon[33714]: osdmap e823: 8 total, 8 up, 8 in 2024-03-24T12:14:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:08 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1355229407' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:08 smithi119 ceph-mon[33714]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:09.223 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:09.224+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v824) v1 ==== 221+0+0 (unknown 2338343215 0 0) 0x7f4ad0053c50 con 0x7f4ae0108dc0 2024-03-24T12:14:09.330 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:09.331+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f4acc000c40 con 0x7f4ae0108dc0 2024-03-24T12:14:10.229 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:10.229+0000 7f4ac7fff640 1 -- 172.21.15.114:0/1355229407 <== mon.2 v1:172.21.15.114:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v825) v1 ==== 221+0+0 (unknown 187734222 0 0) 0x7f4ad0055470 con 0x7f4ae0108dc0 2024-03-24T12:14:10.229 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_bytes = 100 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:14:10.232 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:10.232+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 >> v1:172.21.15.114:6800/1253397276 conn(0x7f4ac8070790 legacy=0x7f4ac8072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:10.232 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:10.232+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 >> v1:172.21.15.114:6790/0 conn(0x7f4ae0108dc0 legacy=0x7f4ae019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:10.232 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:10.232+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 shutdown_connections 2024-03-24T12:14:10.232 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:10.232+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 >> 172.21.15.114:0/1355229407 conn(0x7f4ae00fb980 msgr2=0x7f4ae010f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:10.232 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:10.232+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 shutdown_connections 2024-03-24T12:14:10.232 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:10.232+0000 7f4ae5165640 1 -- 172.21.15.114:0/1355229407 wait complete. 2024-03-24T12:14:10.243 INFO:tasks.workunit.client.0.smithi114.stderr:+ sleep 30 2024-03-24T12:14:10.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[30312]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]': finished 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[30312]: osdmap e824: 8 total, 8 up, 8 in 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1355229407' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[30312]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[30312]: pgmap v1782: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[42466]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]': finished 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[42466]: osdmap e824: 8 total, 8 up, 8 in 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1355229407' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[42466]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:10.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:10 smithi114 ceph-mon[42466]: pgmap v1782: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:10 smithi119 ceph-mon[33714]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]': finished 2024-03-24T12:14:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:10 smithi119 ceph-mon[33714]: osdmap e824: 8 total, 8 up, 8 in 2024-03-24T12:14:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:10 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1355229407' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:10 smithi119 ceph-mon[33714]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-24T12:14:10.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:10 smithi119 ceph-mon[33714]: pgmap v1782: 176 pgs: 176 active+clean; 698 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:11.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:11 smithi114 ceph-mon[42466]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]': finished 2024-03-24T12:14:11.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:11 smithi114 ceph-mon[42466]: osdmap e825: 8 total, 8 up, 8 in 2024-03-24T12:14:11.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:11 smithi114 ceph-mon[30312]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]': finished 2024-03-24T12:14:11.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:11 smithi114 ceph-mon[30312]: osdmap e825: 8 total, 8 up, 8 in 2024-03-24T12:14:11.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:11 smithi119 ceph-mon[33714]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "100"}]': finished 2024-03-24T12:14:11.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:11 smithi119 ceph-mon[33714]: osdmap e825: 8 total, 8 up, 8 in 2024-03-24T12:14:12.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:12 smithi114 ceph-mon[30312]: pgmap v1784: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 5.0 KiB/s wr, 1 op/s 2024-03-24T12:14:12.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:12 smithi114 ceph-mon[42466]: pgmap v1784: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 5.0 KiB/s wr, 1 op/s 2024-03-24T12:14:12.552 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:12 smithi119 ceph-mon[33714]: pgmap v1784: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 5.0 KiB/s wr, 1 op/s 2024-03-24T12:14:13.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:13 smithi114 ceph-mon[30312]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' is full (reached quota's max_bytes: 100 B) 2024-03-24T12:14:13.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:13 smithi114 ceph-mon[30312]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:14:13.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:13 smithi114 ceph-mon[30312]: osdmap e826: 8 total, 8 up, 8 in 2024-03-24T12:14:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:13 smithi114 ceph-mon[42466]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' is full (reached quota's max_bytes: 100 B) 2024-03-24T12:14:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:13 smithi114 ceph-mon[42466]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:14:13.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:13 smithi114 ceph-mon[42466]: osdmap e826: 8 total, 8 up, 8 in 2024-03-24T12:14:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:13 smithi119 ceph-mon[33714]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' is full (reached quota's max_bytes: 100 B) 2024-03-24T12:14:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:13 smithi119 ceph-mon[33714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:14:13.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:13 smithi119 ceph-mon[33714]: osdmap e826: 8 total, 8 up, 8 in 2024-03-24T12:14:13.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:14:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:14:14.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[30312]: pgmap v1786: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[42466]: pgmap v1786: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:14:14.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:14:14.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:14:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:14:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:14 smithi119 ceph-mon[33714]: pgmap v1786: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-24T12:14:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:14:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:14:14.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:14:14.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:14:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:14:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:16.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:16 smithi119 ceph-mon[33714]: pgmap v1787: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 821 B/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-24T12:14:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:16 smithi114 ceph-mon[30312]: pgmap v1787: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 821 B/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-24T12:14:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:16 smithi114 ceph-mon[42466]: pgmap v1787: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 821 B/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-24T12:14:18.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:18 smithi119 ceph-mon[33714]: pgmap v1788: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-24T12:14:18.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:18 smithi114 ceph-mon[42466]: pgmap v1788: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-24T12:14:18.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:18 smithi114 ceph-mon[30312]: pgmap v1788: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-24T12:14:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:20 smithi119 ceph-mon[33714]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-03-24T12:14:20.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:20 smithi114 ceph-mon[30312]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-03-24T12:14:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:20 smithi114 ceph-mon[42466]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-03-24T12:14:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:22.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:22 smithi119 ceph-mon[33714]: pgmap v1790: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:22 smithi114 ceph-mon[30312]: pgmap v1790: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:22.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:22 smithi114 ceph-mon[42466]: pgmap v1790: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:23.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:14:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:14:24.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:24.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:24 smithi114 ceph-mon[30312]: pgmap v1791: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-24T12:14:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:24.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:24 smithi114 ceph-mon[42466]: pgmap v1791: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-24T12:14:24.552 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:24.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:24 smithi119 ceph-mon[33714]: pgmap v1791: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-24T12:14:24.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:14:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:14:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:26.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:26 smithi119 ceph-mon[33714]: pgmap v1792: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:26.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:26 smithi114 ceph-mon[30312]: pgmap v1792: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:26 smithi114 ceph-mon[42466]: pgmap v1792: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:28 smithi119 ceph-mon[33714]: pgmap v1793: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:28 smithi114 ceph-mon[30312]: pgmap v1793: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:28 smithi114 ceph-mon[42466]: pgmap v1793: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:30.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:30 smithi119 ceph-mon[33714]: pgmap v1794: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:30 smithi114 ceph-mon[30312]: pgmap v1794: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:30.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:30 smithi114 ceph-mon[42466]: pgmap v1794: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:32.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:32 smithi114 ceph-mon[30312]: pgmap v1795: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:32.789 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:32 smithi114 ceph-mon[42466]: pgmap v1795: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:32.806 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:32 smithi119 ceph-mon[33714]: pgmap v1795: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:33.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:14:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:14:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:34.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:34 smithi119 ceph-mon[33714]: pgmap v1796: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:34 smithi114 ceph-mon[30312]: pgmap v1796: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:34 smithi114 ceph-mon[42466]: pgmap v1796: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:14:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:14:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:35.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:35 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:35.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:35 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:35.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:35 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:36.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:36 smithi119 ceph-mon[33714]: pgmap v1797: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:36.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:36 smithi114 ceph-mon[30312]: pgmap v1797: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:36.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:36 smithi114 ceph-mon[42466]: pgmap v1797: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:38.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:38 smithi119 ceph-mon[33714]: pgmap v1798: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:38 smithi114 ceph-mon[30312]: pgmap v1798: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:38.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:38 smithi114 ceph-mon[42466]: pgmap v1798: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:40.244 INFO:tasks.workunit.client.0.smithi114.stderr:+ pid=159727 2024-03-24T12:14:40.245 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 5e219aa2-0962-4332-a6f6-f7f15fae1d00 max_bytes 0 2024-03-24T12:14:40.245 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put two /etc/passwd 2024-03-24T12:14:40.342 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.340+0000 7f2811930640 1 -- 172.21.15.114:0/3394908111 >> v1:172.21.15.114:6789/0 conn(0x7f280c0684d0 legacy=0x7f280c0688b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:40.342 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.340+0000 7f2811930640 1 -- 172.21.15.114:0/3394908111 shutdown_connections 2024-03-24T12:14:40.342 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.340+0000 7f2811930640 1 -- 172.21.15.114:0/3394908111 >> 172.21.15.114:0/3394908111 conn(0x7f280c075a30 msgr2=0x7f280c075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:40.342 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.340+0000 7f2811930640 1 -- 172.21.15.114:0/3394908111 shutdown_connections 2024-03-24T12:14:40.342 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.340+0000 7f2811930640 1 -- 172.21.15.114:0/3394908111 wait complete. 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.340+0000 7f2811930640 1 Processor -- start 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.340+0000 7f2811930640 1 -- start start 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f2811930640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f280c1a5e30 con 0x7f280c109f00 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f2811930640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f280c1a7030 con 0x7f280c0684d0 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f2811930640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f280c1a8230 con 0x7f280c10da40 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f2803fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f280c109f00 0x7f280c1a0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35908/0 (socket says 172.21.15.114:35908) 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f2803fff640 1 -- 172.21.15.114:0/3223490423 learned_addr learned my addr 172.21.15.114:0/3223490423 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131580628 0 0) 0x7f280c1a8230 con 0x7f280c10da40 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f8003160 con 0x7f280c10da40 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274692234 0 0) 0x7f280c1a5e30 con 0x7f280c109f00 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f280c1a8230 con 0x7f280c109f00 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212484786 0 0) 0x7f280c1a7030 con 0x7f280c0684d0 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f280c1a5e30 con 0x7f280c0684d0 2024-03-24T12:14:40.343 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1386645215 0 0) 0x7f27f8003160 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f280c1a7030 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077585566 0 0) 0x7f280c1a8230 con 0x7f280c109f00 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27f8003160 con 0x7f280c109f00 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2804001ae0 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206165624 0 0) 0x7f280c1a5e30 con 0x7f280c0684d0 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f280c1a8230 con 0x7f280c0684d0 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f27f40018d0 con 0x7f280c109f00 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f27fc004360 con 0x7f280c0684d0 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495150735 0 0) 0x7f280c1a7030 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 >> v1:172.21.15.119:6789/0 conn(0x7f280c0684d0 legacy=0x7f280c1063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 >> v1:172.21.15.114:6789/0 conn(0x7f280c109f00 legacy=0x7f280c1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c1a9430 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.341+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c1a8420 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.342+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c1a8930 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.342+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2804001cd0 con 0x7f280c10da40 2024-03-24T12:14:40.344 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.342+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2804004c50 con 0x7f280c10da40 2024-03-24T12:14:40.345 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.342+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec005080 con 0x7f280c10da40 2024-03-24T12:14:40.345 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.343+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f2804003970 con 0x7f280c10da40 2024-03-24T12:14:40.345 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.343+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(826..826 src has 254..826) v4 ==== 7802+0+0 (unknown 1172030096 0 0) 0x7f28040879c0 con 0x7f280c10da40 2024-03-24T12:14:40.345 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.344+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=827}) v3 -- 0x7f280c1a7030 con 0x7f280c10da40 2024-03-24T12:14:40.347 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.348+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f2804054650 con 0x7f280c10da40 2024-03-24T12:14:40.525 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.526+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f27ec005330 con 0x7f280c10da40 2024-03-24T12:14:40.537 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.538+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 10 ==== osd_map(827..827 src has 254..827) v4 ==== 628+0+0 (unknown 676130077 0 0) 0x7f2804055210 con 0x7f280c10da40 2024-03-24T12:14:40.537 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.538+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=828}) v3 -- 0x7f27f8003160 con 0x7f280c10da40 2024-03-24T12:14:40.542 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.543+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v827) v1 ==== 217+0+0 (unknown 2873961202 0 0) 0x7f2804052c50 con 0x7f280c10da40 2024-03-24T12:14:40.648 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:40.649+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f27ec000c40 con 0x7f280c10da40 2024-03-24T12:14:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:40 smithi119 ceph-mon[33714]: pgmap v1799: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:40 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3223490423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:40.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:40 smithi119 ceph-mon[33714]: from='client.56384 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:40 smithi114 ceph-mon[30312]: pgmap v1799: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:40 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3223490423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:40 smithi114 ceph-mon[30312]: from='client.56384 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:40 smithi114 ceph-mon[42466]: pgmap v1799: 176 pgs: 176 active+clean; 721 KiB data, 909 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:14:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:40 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3223490423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:40 smithi114 ceph-mon[42466]: from='client.56384 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:41.550 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.550+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 12 ==== osd_map(828..828 src has 254..828) v4 ==== 628+0+0 (unknown 449702879 0 0) 0x7f2804058c60 con 0x7f280c10da40 2024-03-24T12:14:41.550 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.551+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=829}) v3 -- 0x7f280c1a5e30 con 0x7f280c10da40 2024-03-24T12:14:41.554 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.555+0000 7f281092e640 1 -- 172.21.15.114:0/3223490423 <== mon.2 v1:172.21.15.114:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v828) v1 ==== 217+0+0 (unknown 2517952122 0 0) 0x7f2804054470 con 0x7f280c10da40 2024-03-24T12:14:41.554 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_bytes = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:14:41.558 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.559+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 >> v1:172.21.15.114:6800/1253397276 conn(0x7f27f806e4e0 legacy=0x7f27f80709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:41.558 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.559+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 >> v1:172.21.15.114:6790/0 conn(0x7f280c10da40 legacy=0x7f280c1a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:41.558 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.559+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 shutdown_connections 2024-03-24T12:14:41.558 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.559+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 >> 172.21.15.114:0/3223490423 conn(0x7f280c075a30 msgr2=0x7f280c111230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:41.558 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.559+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 shutdown_connections 2024-03-24T12:14:41.558 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.559+0000 7f2811930640 1 -- 172.21.15.114:0/3223490423 wait complete. 2024-03-24T12:14:41.569 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 5e219aa2-0962-4332-a6f6-f7f15fae1d00 max_objects 0 2024-03-24T12:14:41.657 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.655+0000 7f2e9f113640 1 -- 172.21.15.114:0/226567029 >> v1:172.21.15.114:6789/0 conn(0x7f2e980ff9f0 legacy=0x7f2e980ffdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:41.657 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- 172.21.15.114:0/226567029 shutdown_connections 2024-03-24T12:14:41.657 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- 172.21.15.114:0/226567029 >> 172.21.15.114:0/226567029 conn(0x7f2e98075210 msgr2=0x7f2e98075620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:41.657 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- 172.21.15.114:0/226567029 shutdown_connections 2024-03-24T12:14:41.658 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- 172.21.15.114:0/226567029 wait complete. 2024-03-24T12:14:41.658 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 Processor -- start 2024-03-24T12:14:41.658 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- start start 2024-03-24T12:14:41.658 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e981a8000 con 0x7f2e980ff9f0 2024-03-24T12:14:41.658 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e981a9200 con 0x7f2e98109ee0 2024-03-24T12:14:41.658 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9f113640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e981aa400 con 0x7f2e9810da50 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9ceaf640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2e980ff9f0 0x7f2e981063f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33250/0 (socket says 172.21.15.114:33250) 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.656+0000 7f2e9ceaf640 1 -- 172.21.15.114:0/2937418175 learned_addr learned my addr 172.21.15.114:0/2937418175 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588397505 0 0) 0x7f2e981a8000 con 0x7f2e980ff9f0 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e88003160 con 0x7f2e980ff9f0 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314098321 0 0) 0x7f2e981aa400 con 0x7f2e9810da50 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e981a8000 con 0x7f2e9810da50 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604102464 0 0) 0x7f2e981a9200 con 0x7f2e98109ee0 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e981aa400 con 0x7f2e98109ee0 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1630461588 0 0) 0x7f2e88003160 con 0x7f2e980ff9f0 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e981a9200 con 0x7f2e980ff9f0 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2e8c004510 con 0x7f2e980ff9f0 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1228127211 0 0) 0x7f2e981a8000 con 0x7f2e9810da50 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e88003160 con 0x7f2e9810da50 2024-03-24T12:14:41.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2e900013f0 con 0x7f2e9810da50 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847012583 0 0) 0x7f2e981aa400 con 0x7f2e98109ee0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e981a8000 con 0x7f2e98109ee0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2e84001870 con 0x7f2e98109ee0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3687845552 0 0) 0x7f2e981a9200 con 0x7f2e980ff9f0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 >> v1:172.21.15.114:6790/0 conn(0x7f2e9810da50 legacy=0x7f2e981a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 >> v1:172.21.15.119:6789/0 conn(0x7f2e98109ee0 legacy=0x7f2e981a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e981ab600 con 0x7f2e980ff9f0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e981aa5f0 con 0x7f2e980ff9f0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2e8c003bf0 con 0x7f2e980ff9f0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2e8c005110 con 0x7f2e980ff9f0 2024-03-24T12:14:41.660 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.657+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e981aaa80 con 0x7f2e980ff9f0 2024-03-24T12:14:41.661 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.658+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980692e0 con 0x7f2e980ff9f0 2024-03-24T12:14:41.661 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.659+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f2e8c003e60 con 0x7f2e980ff9f0 2024-03-24T12:14:41.661 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.659+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(828..828 src has 254..828) v4 ==== 7802+0+0 (unknown 972193000 0 0) 0x7f2e8c088bb0 con 0x7f2e980ff9f0 2024-03-24T12:14:41.661 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.660+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=829}) v3 -- 0x7f2e981a9200 con 0x7f2e980ff9f0 2024-03-24T12:14:41.663 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.664+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f2e8c055bd0 con 0x7f2e980ff9f0 2024-03-24T12:14:41.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:41 smithi119 ceph-mon[33714]: from='client.56384 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:14:41.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:41 smithi119 ceph-mon[33714]: osdmap e827: 8 total, 8 up, 8 in 2024-03-24T12:14:41.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:41 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3223490423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:41.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:41 smithi119 ceph-mon[33714]: from='client.56384 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:41.837 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:41.838+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f2e98101e40 con 0x7f2e980ff9f0 2024-03-24T12:14:41.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[30312]: from='client.56384 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:14:41.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[30312]: osdmap e827: 8 total, 8 up, 8 in 2024-03-24T12:14:41.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3223490423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:41.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[30312]: from='client.56384 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:41.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[42466]: from='client.56384 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:14:41.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[42466]: osdmap e827: 8 total, 8 up, 8 in 2024-03-24T12:14:41.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3223490423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:41.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:41 smithi114 ceph-mon[42466]: from='client.56384 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:14:42.550 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:42.551+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v829) v1 ==== 221+0+0 (unknown 1720824390 0 0) 0x7f2e8c053f40 con 0x7f2e980ff9f0 2024-03-24T12:14:42.552 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:42.554+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 11 ==== osd_map(829..829 src has 254..829) v4 ==== 628+0+0 (unknown 3697095273 0 0) 0x7f2e8c056500 con 0x7f2e980ff9f0 2024-03-24T12:14:42.553 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:42.554+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=830}) v3 -- 0x7f2e88003160 con 0x7f2e980ff9f0 2024-03-24T12:14:42.663 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:42.664+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f2e981137a0 con 0x7f2e980ff9f0 2024-03-24T12:14:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[30312]: pgmap v1801: 176 pgs: 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[30312]: from='client.56384 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:14:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[30312]: osdmap e828: 8 total, 8 up, 8 in 2024-03-24T12:14:42.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:14:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[42466]: pgmap v1801: 176 pgs: 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[42466]: from='client.56384 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:14:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[42466]: osdmap e828: 8 total, 8 up, 8 in 2024-03-24T12:14:42.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:42 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:14:43.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:42 smithi119 ceph-mon[33714]: pgmap v1801: 176 pgs: 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:43.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:42 smithi119 ceph-mon[33714]: from='client.56384 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:14:43.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:42 smithi119 ceph-mon[33714]: osdmap e828: 8 total, 8 up, 8 in 2024-03-24T12:14:43.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:42 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:14:43.226 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.227+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v830) v1 ==== 221+0+0 (unknown 3817649812 0 0) 0x7f2e8c055760 con 0x7f2e980ff9f0 2024-03-24T12:14:43.226 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_objects = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:14:43.227 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.228+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 <== mon.0 v1:172.21.15.114:6789/0 13 ==== osd_map(830..830 src has 254..830) v4 ==== 628+0+0 (unknown 2610142775 0 0) 0x7f2e8c059f50 con 0x7f2e980ff9f0 2024-03-24T12:14:43.227 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.229+0000 7f2e95ffb640 1 -- 172.21.15.114:0/2937418175 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=831}) v3 -- 0x7f2e981aa400 con 0x7f2e980ff9f0 2024-03-24T12:14:43.230 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.231+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 >> v1:172.21.15.114:6800/1253397276 conn(0x7f2e8806e5d0 legacy=0x7f2e88070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:43.230 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.231+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 >> v1:172.21.15.114:6789/0 conn(0x7f2e980ff9f0 legacy=0x7f2e981063f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:43.230 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.231+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 shutdown_connections 2024-03-24T12:14:43.230 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.231+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 >> 172.21.15.114:0/2937418175 conn(0x7f2e98075210 msgr2=0x7f2e980fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:43.230 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.231+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 shutdown_connections 2024-03-24T12:14:43.231 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.231+0000 7f2e9f113640 1 -- 172.21.15.114:0/2937418175 wait complete. 2024-03-24T12:14:43.245 INFO:tasks.workunit.client.0.smithi114.stderr:+ wait 159727 2024-03-24T12:14:43.245 INFO:tasks.workunit.client.0.smithi114.stderr:+ '[' 0 -ne 0 ']' 2024-03-24T12:14:43.245 INFO:tasks.workunit.client.0.smithi114.stderr:+ true 2024-03-24T12:14:43.245 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put three /etc/passwd 2024-03-24T12:14:43.280 INFO:tasks.workunit.client.0.smithi114.stderr:++ uuidgen 2024-03-24T12:14:43.281 INFO:tasks.workunit.client.0.smithi114.stderr:+ pp=12c1aa96-773c-4a69-921d-e5e47a731694 2024-03-24T12:14:43.281 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool create 12c1aa96-773c-4a69-921d-e5e47a731694 12 2024-03-24T12:14:43.364 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.365+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2392116468 >> v1:172.21.15.114:6789/0 conn(0x7fc09c10c950 legacy=0x7fc09c10ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:43.364 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.365+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2392116468 shutdown_connections 2024-03-24T12:14:43.364 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.365+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2392116468 >> 172.21.15.114:0/2392116468 conn(0x7fc09c075210 msgr2=0x7fc09c075620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:43.364 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.365+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2392116468 shutdown_connections 2024-03-24T12:14:43.364 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.365+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2392116468 wait complete. 2024-03-24T12:14:43.365 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.365+0000 7fc0a1a88640 1 Processor -- start 2024-03-24T12:14:43.365 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.365+0000 7fc0a1a88640 1 -- start start 2024-03-24T12:14:43.367 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc0a1a88640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc09c1a7fa0 con 0x7fc09c108de0 2024-03-24T12:14:43.367 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc0a1a88640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc09c1a91a0 con 0x7fc09c10c950 2024-03-24T12:14:43.367 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc0a1a88640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc09c1aa3a0 con 0x7fc09c100b00 2024-03-24T12:14:43.367 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc09b7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc09c100b00 0x7fc09c105390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58010/0 (socket says 172.21.15.114:58010) 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc09affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc09c108de0 0x7fc09c1a30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33314/0 (socket says 172.21.15.114:33314) 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc09b7fe640 1 -- 172.21.15.114:0/2715211590 learned_addr learned my addr 172.21.15.114:0/2715211590 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052991125 0 0) 0x7fc09c1a7fa0 con 0x7fc09c108de0 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc084003520 con 0x7fc09c108de0 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000076945 0 0) 0x7fc09c1aa3a0 con 0x7fc09c100b00 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc09c1a7fa0 con 0x7fc09c100b00 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014086762 0 0) 0x7fc09c1a91a0 con 0x7fc09c10c950 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc09c1aa3a0 con 0x7fc09c10c950 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3470278430 0 0) 0x7fc084003520 con 0x7fc09c108de0 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc09c1a91a0 con 0x7fc09c108de0 2024-03-24T12:14:43.368 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156495041 0 0) 0x7fc09c1a7fa0 con 0x7fc09c100b00 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc084003520 con 0x7fc09c100b00 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fc090001630 con 0x7fc09c108de0 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fc08c0018f0 con 0x7fc09c100b00 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180912761 0 0) 0x7fc09c1aa3a0 con 0x7fc09c10c950 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc09c1a7fa0 con 0x7fc09c10c950 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fc0940046d0 con 0x7fc09c10c950 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1211922814 0 0) 0x7fc09c1a91a0 con 0x7fc09c108de0 2024-03-24T12:14:43.369 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 >> v1:172.21.15.114:6790/0 conn(0x7fc09c100b00 legacy=0x7fc09c105390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 >> v1:172.21.15.119:6789/0 conn(0x7fc09c10c950 legacy=0x7fc09c1a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c1ab5a0 con 0x7fc09c108de0 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c1a8190 con 0x7fc09c108de0 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.366+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c1a8620 con 0x7fc09c108de0 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.367+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fc090003a80 con 0x7fc09c108de0 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.367+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fc090005a90 con 0x7fc09c108de0 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.367+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc088005080 con 0x7fc09c108de0 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.368+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7fc0900035f0 con 0x7fc09c108de0 2024-03-24T12:14:43.370 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.368+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(830..830 src has 254..830) v4 ==== 7802+0+0 (unknown 303371968 0 0) 0x7fc090088860 con 0x7fc09c108de0 2024-03-24T12:14:43.372 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.373+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fc0900557a0 con 0x7fc09c108de0 2024-03-24T12:14:43.548 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:43.549+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12} v 0) v1 -- 0x7fc0880058b0 con 0x7fc09c108de0 2024-03-24T12:14:43.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:14:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:14:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-mon[33714]: osdmap e829: 8 total, 8 up, 8 in 2024-03-24T12:14:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:14:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-mon[33714]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' no longer out of quota; removing NO_QUOTA flag 2024-03-24T12:14:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-mon[33714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:14:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:14:44.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:43 smithi119 ceph-mon[33714]: osdmap e830: 8 total, 8 up, 8 in 2024-03-24T12:14:44.233 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.234+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]=0 pool '12c1aa96-773c-4a69-921d-e5e47a731694' created v831) v1 ==== 176+0+0 (unknown 2925637295 0 0) 0x7fc090053b10 con 0x7fc09c108de0 2024-03-24T12:14:44.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:14:44.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[30312]: osdmap e829: 8 total, 8 up, 8 in 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[30312]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' no longer out of quota; removing NO_QUOTA flag 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[30312]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[30312]: osdmap e830: 8 total, 8 up, 8 in 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[42466]: osdmap e829: 8 total, 8 up, 8 in 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[42466]: pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' no longer out of quota; removing NO_QUOTA flag 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[42466]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2937418175' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:14:44.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:43 smithi114 ceph-mon[42466]: osdmap e830: 8 total, 8 up, 8 in 2024-03-24T12:14:44.347 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.348+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12} v 0) v1 -- 0x7fc0880053e0 con 0x7fc09c108de0 2024-03-24T12:14:44.347 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.348+0000 7fc098ff9640 1 -- 172.21.15.114:0/2715211590 <== mon.0 v1:172.21.15.114:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]=0 pool '12c1aa96-773c-4a69-921d-e5e47a731694' already exists v831) v1 ==== 183+0+0 (unknown 4139082376 0 0) 0x7fc0880053e0 con 0x7fc09c108de0 2024-03-24T12:14:44.347 INFO:tasks.workunit.client.0.smithi114.stderr:pool '12c1aa96-773c-4a69-921d-e5e47a731694' already exists 2024-03-24T12:14:44.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.350+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 >> v1:172.21.15.114:6800/1253397276 conn(0x7fc08406e730 legacy=0x7fc084070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:44.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.350+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 >> v1:172.21.15.114:6789/0 conn(0x7fc09c108de0 legacy=0x7fc09c1a30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:44.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.350+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 shutdown_connections 2024-03-24T12:14:44.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.350+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 >> 172.21.15.114:0/2715211590 conn(0x7fc09c075210 msgr2=0x7fc09c0679d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:44.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.350+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 shutdown_connections 2024-03-24T12:14:44.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.350+0000 7fc0a1a88640 1 -- 172.21.15.114:0/2715211590 wait complete. 2024-03-24T12:14:44.358 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool application enable 12c1aa96-773c-4a69-921d-e5e47a731694 rados 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.443+0000 7fb84b40e640 1 -- 172.21.15.114:0/4077429248 >> v1:172.21.15.114:6789/0 conn(0x7fb844100280 legacy=0x7fb844100660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.443+0000 7fb84b40e640 1 -- 172.21.15.114:0/4077429248 shutdown_connections 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.443+0000 7fb84b40e640 1 -- 172.21.15.114:0/4077429248 >> 172.21.15.114:0/4077429248 conn(0x7fb844075a30 msgr2=0x7fb844075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.443+0000 7fb84b40e640 1 -- 172.21.15.114:0/4077429248 shutdown_connections 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.443+0000 7fb84b40e640 1 -- 172.21.15.114:0/4077429248 wait complete. 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb84b40e640 1 Processor -- start 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb84b40e640 1 -- start start 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb84b40e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb8441a80d0 con 0x7fb84410c930 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb84b40e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb8441a92d0 con 0x7fb844100280 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb84b40e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb8441aa4d0 con 0x7fb844108dc0 2024-03-24T12:14:44.445 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8489a9640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fb844108dc0 0x7fb8441a30a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58020/0 (socket says 172.21.15.114:58020) 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8489a9640 1 -- 172.21.15.114:0/448888617 learned_addr learned my addr 172.21.15.114:0/448888617 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055416725 0 0) 0x7fb8441a80d0 con 0x7fb84410c930 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c003520 con 0x7fb84410c930 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193147432 0 0) 0x7fb8441aa4d0 con 0x7fb844108dc0 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8441a80d0 con 0x7fb844108dc0 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118307409 0 0) 0x7fb8441a92d0 con 0x7fb844100280 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8441aa4d0 con 0x7fb844100280 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2193525404 0 0) 0x7fb82c003520 con 0x7fb84410c930 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8441a92d0 con 0x7fb84410c930 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb834001c30 con 0x7fb84410c930 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908197143 0 0) 0x7fb8441a80d0 con 0x7fb844108dc0 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82c003520 con 0x7fb844108dc0 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb838001900 con 0x7fb844108dc0 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198505304 0 0) 0x7fb8441aa4d0 con 0x7fb844100280 2024-03-24T12:14:44.446 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8441a80d0 con 0x7fb844100280 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.444+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb83c004430 con 0x7fb844100280 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3463817074 0 0) 0x7fb8441a92d0 con 0x7fb84410c930 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 >> v1:172.21.15.114:6790/0 conn(0x7fb844108dc0 legacy=0x7fb8441a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 >> v1:172.21.15.119:6789/0 conn(0x7fb844100280 legacy=0x7fb844105350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8441ab6d0 con 0x7fb84410c930 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8441a82c0 con 0x7fb84410c930 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8441a8750 con 0x7fb84410c930 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fb834003e20 con 0x7fb84410c930 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.445+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fb834004c50 con 0x7fb84410c930 2024-03-24T12:14:44.447 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.446+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb830005080 con 0x7fb84410c930 2024-03-24T12:14:44.448 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.446+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7fb834003990 con 0x7fb84410c930 2024-03-24T12:14:44.448 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.447+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(831..831 src has 254..831) v4 ==== 8177+0+0 (unknown 1249786195 0 0) 0x7fb834087ad0 con 0x7fb84410c930 2024-03-24T12:14:44.450 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.451+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fb834054830 con 0x7fb84410c930 2024-03-24T12:14:44.622 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:44.623+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"} v 0) v1 -- 0x7fb830005330 con 0x7fb84410c930 2024-03-24T12:14:44.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:14:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[30312]: pgmap v1805: 176 pgs: 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]: dispatch 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]': finished 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[30312]: osdmap e831: 8 total, 8 up, 8 in 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]: dispatch 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]: dispatch 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[42466]: pgmap v1805: 176 pgs: 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]: dispatch 2024-03-24T12:14:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]': finished 2024-03-24T12:14:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[42466]: osdmap e831: 8 total, 8 up, 8 in 2024-03-24T12:14:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]: dispatch 2024-03-24T12:14:44.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:44 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]: dispatch 2024-03-24T12:14:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:44 smithi119 ceph-mon[33714]: pgmap v1805: 176 pgs: 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:14:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]: dispatch 2024-03-24T12:14:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]': finished 2024-03-24T12:14:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:44 smithi119 ceph-mon[33714]: osdmap e831: 8 total, 8 up, 8 in 2024-03-24T12:14:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2715211590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pg_num": 12}]: dispatch 2024-03-24T12:14:45.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:44 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]: dispatch 2024-03-24T12:14:45.235 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:45.236+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]=0 enabled application 'rados' on pool '12c1aa96-773c-4a69-921d-e5e47a731694' v832) v1 ==== 213+0+0 (unknown 1447657320 0 0) 0x7fb834052ba0 con 0x7fb84410c930 2024-03-24T12:14:45.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:45.351+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"} v 0) v1 -- 0x7fb830001090 con 0x7fb84410c930 2024-03-24T12:14:46.241 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.242+0000 7fb8427fc640 1 -- 172.21.15.114:0/448888617 <== mon.0 v1:172.21.15.114:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]=0 enabled application 'rados' on pool '12c1aa96-773c-4a69-921d-e5e47a731694' v833) v1 ==== 213+0+0 (unknown 1818790286 0 0) 0x7fb8340543c0 con 0x7fb84410c930 2024-03-24T12:14:46.241 INFO:tasks.workunit.client.0.smithi114.stderr:enabled application 'rados' on pool '12c1aa96-773c-4a69-921d-e5e47a731694' 2024-03-24T12:14:46.245 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.246+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 >> v1:172.21.15.114:6800/1253397276 conn(0x7fb82c06e580 legacy=0x7fb82c070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:46.245 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.246+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 >> v1:172.21.15.114:6789/0 conn(0x7fb84410c930 legacy=0x7fb8441a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:46.245 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.246+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 shutdown_connections 2024-03-24T12:14:46.245 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.246+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 >> 172.21.15.114:0/448888617 conn(0x7fb844075a30 msgr2=0x7fb8441100a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:46.246 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.247+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 shutdown_connections 2024-03-24T12:14:46.246 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.247+0000 7fb84b40e640 1 -- 172.21.15.114:0/448888617 wait complete. 2024-03-24T12:14:46.258 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 12c1aa96-773c-4a69-921d-e5e47a731694 max_objects 10 2024-03-24T12:14:46.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.347+0000 7fecbe65b640 1 -- 172.21.15.114:0/4081681670 >> v1:172.21.15.114:6789/0 conn(0x7fecb8109ed0 legacy=0x7fecb810c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:46.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.347+0000 7fecbe65b640 1 -- 172.21.15.114:0/4081681670 shutdown_connections 2024-03-24T12:14:46.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.347+0000 7fecbe65b640 1 -- 172.21.15.114:0/4081681670 >> 172.21.15.114:0/4081681670 conn(0x7fecb80fcaf0 msgr2=0x7fecb80fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:46.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.347+0000 7fecbe65b640 1 -- 172.21.15.114:0/4081681670 shutdown_connections 2024-03-24T12:14:46.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.347+0000 7fecbe65b640 1 -- 172.21.15.114:0/4081681670 wait complete. 2024-03-24T12:14:46.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecbe65b640 1 Processor -- start 2024-03-24T12:14:46.349 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecbe65b640 1 -- start start 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecbe65b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecb81a5b70 con 0x7fecb8101250 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecbe65b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecb81a6d70 con 0x7fecb810da40 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecbe65b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecb81a7f70 con 0x7fecb8109ed0 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecb7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fecb8101250 0x7fecb8103410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33354/0 (socket says 172.21.15.114:33354) 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecb7fff640 1 -- 172.21.15.114:0/2246810674 learned_addr learned my addr 172.21.15.114:0/2246810674 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162953400 0 0) 0x7fecb81a5b70 con 0x7fecb8101250 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca0001fe0 con 0x7fecb8101250 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777312844 0 0) 0x7fecb81a7f70 con 0x7fecb8109ed0 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.348+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb81a5b70 con 0x7fecb8109ed0 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2726554302 0 0) 0x7fecb81a5b70 con 0x7fecb8109ed0 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecb81a7f70 con 0x7fecb8109ed0 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fecac0015d0 con 0x7fecb8109ed0 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761090852 0 0) 0x7feca0001fe0 con 0x7fecb8101250 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecb81a5b70 con 0x7fecb8101250 2024-03-24T12:14:46.350 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7feca8001940 con 0x7fecb8101250 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018626043 0 0) 0x7fecb81a6d70 con 0x7fecb810da40 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca0001fe0 con 0x7fecb810da40 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2825415658 0 0) 0x7fecb81a7f70 con 0x7fecb8109ed0 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 >> v1:172.21.15.119:6789/0 conn(0x7fecb810da40 legacy=0x7fecb81a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 >> v1:172.21.15.114:6789/0 conn(0x7fecb8101250 legacy=0x7fecb8103410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb81a9170 con 0x7fecb8109ed0 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb81a8160 con 0x7fecb8109ed0 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb81a85f0 con 0x7fecb8109ed0 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fecac001a70 con 0x7fecb8109ed0 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.349+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fecac004de0 con 0x7fecb8109ed0 2024-03-24T12:14:46.351 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.350+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca4005070 con 0x7fecb8109ed0 2024-03-24T12:14:46.352 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.350+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7fecac0035f0 con 0x7fecb8109ed0 2024-03-24T12:14:46.352 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.351+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(833..833 src has 254..833) v4 ==== 8190+0+0 (unknown 2937115672 0 0) 0x7fecac0879a0 con 0x7fecb8109ed0 2024-03-24T12:14:46.354 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.355+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fecac054760 con 0x7fecb8109ed0 2024-03-24T12:14:46.534 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:46.535+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7feca4005320 con 0x7fecb8109ed0 2024-03-24T12:14:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]': finished 2024-03-24T12:14:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:46 smithi119 ceph-mon[33714]: osdmap e832: 8 total, 8 up, 8 in 2024-03-24T12:14:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:46 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]: dispatch 2024-03-24T12:14:46.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:46 smithi119 ceph-mon[33714]: pgmap v1808: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:14:46.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]': finished 2024-03-24T12:14:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[30312]: osdmap e832: 8 total, 8 up, 8 in 2024-03-24T12:14:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]: dispatch 2024-03-24T12:14:46.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[30312]: pgmap v1808: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:14:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]': finished 2024-03-24T12:14:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[42466]: osdmap e832: 8 total, 8 up, 8 in 2024-03-24T12:14:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]: dispatch 2024-03-24T12:14:46.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:46 smithi114 ceph-mon[42466]: pgmap v1808: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 910 MiB used, 714 GiB / 715 GiB avail 2024-03-24T12:14:47.246 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:47.247+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 12c1aa96-773c-4a69-921d-e5e47a731694 v834) v1 ==== 223+0+0 (unknown 3844495487 0 0) 0x7fecac052ad0 con 0x7fecb8109ed0 2024-03-24T12:14:47.355 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:47.356+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7feca4001090 con 0x7fecb8109ed0 2024-03-24T12:14:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]': finished 2024-03-24T12:14:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:47 smithi119 ceph-mon[33714]: osdmap e833: 8 total, 8 up, 8 in 2024-03-24T12:14:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:47 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2246810674' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:47.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:47 smithi119 ceph-mon[33714]: from='client.56417 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]': finished 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[30312]: osdmap e833: 8 total, 8 up, 8 in 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2246810674' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[30312]: from='client.56417 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/448888617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "app": "rados"}]': finished 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[42466]: osdmap e833: 8 total, 8 up, 8 in 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2246810674' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:47.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:47 smithi114 ceph-mon[42466]: from='client.56417 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:48.268 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:48.269+0000 7fecb57fa640 1 -- 172.21.15.114:0/2246810674 <== mon.2 v1:172.21.15.114:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 12c1aa96-773c-4a69-921d-e5e47a731694 v835) v1 ==== 223+0+0 (unknown 2892233999 0 0) 0x7fecac0542f0 con 0x7fecb8109ed0 2024-03-24T12:14:48.268 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_objects = 10 for pool 12c1aa96-773c-4a69-921d-e5e47a731694 2024-03-24T12:14:48.271 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:48.272+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 >> v1:172.21.15.114:6800/1253397276 conn(0x7feca006e550 legacy=0x7feca0070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:48.271 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:48.272+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 >> v1:172.21.15.114:6790/0 conn(0x7fecb8109ed0 legacy=0x7fecb8103b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:14:48.271 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:48.272+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 shutdown_connections 2024-03-24T12:14:48.271 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:48.272+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 >> 172.21.15.114:0/2246810674 conn(0x7fecb80fcaf0 msgr2=0x7fecb80fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:14:48.271 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:48.272+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 shutdown_connections 2024-03-24T12:14:48.271 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:14:48.272+0000 7fecbe65b640 1 -- 172.21.15.114:0/2246810674 wait complete. 2024-03-24T12:14:48.282 INFO:tasks.workunit.client.0.smithi114.stderr:+ sleep 30 2024-03-24T12:14:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:48 smithi119 ceph-mon[33714]: from='client.56417 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:14:48.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:48 smithi119 ceph-mon[33714]: osdmap e834: 8 total, 8 up, 8 in 2024-03-24T12:14:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:48 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2246810674' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:48 smithi119 ceph-mon[33714]: from='client.56417 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:48.554 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:48 smithi119 ceph-mon[33714]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-24T12:14:48.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[30312]: from='client.56417 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:14:48.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[30312]: osdmap e834: 8 total, 8 up, 8 in 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2246810674' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[30312]: from='client.56417 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[30312]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[42466]: from='client.56417 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[42466]: osdmap e834: 8 total, 8 up, 8 in 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2246810674' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[42466]: from='client.56417 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]: dispatch 2024-03-24T12:14:48.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:48 smithi114 ceph-mon[42466]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-24T12:14:49.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:49 smithi119 ceph-mon[33714]: from='client.56417 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:14:49.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:49 smithi119 ceph-mon[33714]: osdmap e835: 8 total, 8 up, 8 in 2024-03-24T12:14:49.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:49 smithi114 ceph-mon[42466]: from='client.56417 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:14:49.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:49 smithi114 ceph-mon[42466]: osdmap e835: 8 total, 8 up, 8 in 2024-03-24T12:14:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:49 smithi114 ceph-mon[30312]: from='client.56417 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "field": "max_objects", "val": "10"}]': finished 2024-03-24T12:14:49.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:49 smithi114 ceph-mon[30312]: osdmap e835: 8 total, 8 up, 8 in 2024-03-24T12:14:50.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:50 smithi119 ceph-mon[33714]: pgmap v1813: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2024-03-24T12:14:50.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:50 smithi114 ceph-mon[30312]: pgmap v1813: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2024-03-24T12:14:50.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:50 smithi114 ceph-mon[42466]: pgmap v1813: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2024-03-24T12:14:51.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:51 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:14:51.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:51 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:51.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:51 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:14:51.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:51 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:51.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:51 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:14:51.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:51 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:14:52.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:52 smithi114 ceph-mon[30312]: pgmap v1814: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T12:14:52.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:52 smithi114 ceph-mon[42466]: pgmap v1814: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T12:14:52.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:52 smithi119 ceph-mon[33714]: pgmap v1814: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-24T12:14:53.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:14:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:14:54.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:54.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:54 smithi119 ceph-mon[33714]: pgmap v1815: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.3 KiB/s wr, 1 op/s 2024-03-24T12:14:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:54 smithi114 ceph-mon[30312]: pgmap v1815: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.3 KiB/s wr, 1 op/s 2024-03-24T12:14:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:14:54.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:54 smithi114 ceph-mon[42466]: pgmap v1815: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.3 KiB/s wr, 1 op/s 2024-03-24T12:14:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:14:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:14:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:14:56.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:56 smithi119 ceph-mon[33714]: pgmap v1816: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-03-24T12:14:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:56 smithi114 ceph-mon[30312]: pgmap v1816: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-03-24T12:14:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:56 smithi114 ceph-mon[42466]: pgmap v1816: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-03-24T12:14:58.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:14:58 smithi119 ceph-mon[33714]: pgmap v1817: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:58.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:14:58 smithi114 ceph-mon[30312]: pgmap v1817: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:14:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:14:58 smithi114 ceph-mon[42466]: pgmap v1817: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:15:00.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:00 smithi119 ceph-mon[33714]: pgmap v1818: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-03-24T12:15:00.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:00 smithi114 ceph-mon[30312]: pgmap v1818: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-03-24T12:15:00.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:00 smithi114 ceph-mon[42466]: pgmap v1818: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-03-24T12:15:02.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:02 smithi114 ceph-mon[30312]: pgmap v1819: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:02.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:02 smithi114 ceph-mon[42466]: pgmap v1819: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:02.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:02 smithi119 ceph-mon[33714]: pgmap v1819: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:03.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:15:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:15:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:04.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:04 smithi119 ceph-mon[33714]: pgmap v1820: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:04.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:04 smithi114 ceph-mon[42466]: pgmap v1820: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:04 smithi114 ceph-mon[30312]: pgmap v1820: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:15:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:15:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:05.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:05.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:06.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:06 smithi119 ceph-mon[33714]: pgmap v1821: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:06.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:06 smithi114 ceph-mon[30312]: pgmap v1821: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:06 smithi114 ceph-mon[42466]: pgmap v1821: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:08.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:08 smithi119 ceph-mon[33714]: pgmap v1822: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:08 smithi114 ceph-mon[30312]: pgmap v1822: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:08 smithi114 ceph-mon[42466]: pgmap v1822: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:10.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:10 smithi114 ceph-mon[30312]: pgmap v1823: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:10 smithi114 ceph-mon[42466]: pgmap v1823: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:11.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:10 smithi119 ceph-mon[33714]: pgmap v1823: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:12.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:12 smithi114 ceph-mon[30312]: pgmap v1824: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:12 smithi114 ceph-mon[42466]: pgmap v1824: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:13.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:12 smithi119 ceph-mon[33714]: pgmap v1824: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:13.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:15:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:15:14.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:14 smithi114 ceph-mon[30312]: pgmap v1825: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:14 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:15:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:14 smithi114 ceph-mon[42466]: pgmap v1825: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:14.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:14 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:15:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:15:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:15:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:15.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:14 smithi119 ceph-mon[33714]: pgmap v1825: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:14 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:15:15.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:15:15.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:15:15.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:15:15.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:15:15.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:15:15.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:15:16.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:15:16.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:15:16.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:15:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:16 smithi114 ceph-mon[30312]: pgmap v1826: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:16 smithi114 ceph-mon[42466]: pgmap v1826: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:16 smithi119 ceph-mon[33714]: pgmap v1826: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:18.284 INFO:tasks.workunit.client.0.smithi114.stderr:++ seq 1 10 2024-03-24T12:15:18.284 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.285 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj1 /etc/passwd 2024-03-24T12:15:18.323 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.323 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj2 /etc/passwd 2024-03-24T12:15:18.358 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.358 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj3 /etc/passwd 2024-03-24T12:15:18.392 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.392 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj4 /etc/passwd 2024-03-24T12:15:18.426 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.426 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj5 /etc/passwd 2024-03-24T12:15:18.459 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.459 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj6 /etc/passwd 2024-03-24T12:15:18.491 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.492 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj7 /etc/passwd 2024-03-24T12:15:18.524 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.524 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj8 /etc/passwd 2024-03-24T12:15:18.556 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.556 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj9 /etc/passwd 2024-03-24T12:15:18.589 INFO:tasks.workunit.client.0.smithi114.stderr:+ for f in `seq 1 10` 2024-03-24T12:15:18.589 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 12c1aa96-773c-4a69-921d-e5e47a731694 put obj10 /etc/passwd 2024-03-24T12:15:18.622 INFO:tasks.workunit.client.0.smithi114.stderr:+ sleep 30 2024-03-24T12:15:18.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:18 smithi114 ceph-mon[30312]: pgmap v1827: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:18.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:18 smithi114 ceph-mon[42466]: pgmap v1827: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:19.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:18 smithi119 ceph-mon[33714]: pgmap v1827: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:20 smithi114 ceph-mon[30312]: pgmap v1828: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:20.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:20 smithi114 ceph-mon[42466]: pgmap v1828: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:21.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:20 smithi119 ceph-mon[33714]: pgmap v1828: 188 pgs: 188 active+clean; 744 KiB data, 910 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T12:15:21.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:22 smithi114 ceph-mon[30312]: pgmap v1829: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-24T12:15:22.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:22 smithi114 ceph-mon[42466]: pgmap v1829: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-24T12:15:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:22 smithi119 ceph-mon[33714]: pgmap v1829: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-24T12:15:23.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:15:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:15:23.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:23 smithi114 ceph-mon[30312]: pool '12c1aa96-773c-4a69-921d-e5e47a731694' is full (reached quota's max_objects: 10) 2024-03-24T12:15:23.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:23 smithi114 ceph-mon[30312]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:15:23.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:23 smithi114 ceph-mon[30312]: osdmap e836: 8 total, 8 up, 8 in 2024-03-24T12:15:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:23 smithi114 ceph-mon[42466]: pool '12c1aa96-773c-4a69-921d-e5e47a731694' is full (reached quota's max_objects: 10) 2024-03-24T12:15:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:23 smithi114 ceph-mon[42466]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:15:23.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:23 smithi114 ceph-mon[42466]: osdmap e836: 8 total, 8 up, 8 in 2024-03-24T12:15:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:23 smithi119 ceph-mon[33714]: pool '12c1aa96-773c-4a69-921d-e5e47a731694' is full (reached quota's max_objects: 10) 2024-03-24T12:15:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:23 smithi119 ceph-mon[33714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-24T12:15:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:23 smithi119 ceph-mon[33714]: osdmap e836: 8 total, 8 up, 8 in 2024-03-24T12:15:24.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:24.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:24 smithi114 ceph-mon[42466]: pgmap v1831: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:24.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:24 smithi114 ceph-mon[30312]: pgmap v1831: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:24.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:15:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:15:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:25.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:24 smithi119 ceph-mon[33714]: pgmap v1831: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:26.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:26 smithi114 ceph-mon[30312]: pgmap v1832: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:26.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:26 smithi114 ceph-mon[42466]: pgmap v1832: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:27.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:26 smithi119 ceph-mon[33714]: pgmap v1832: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:28.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:28 smithi114 ceph-mon[30312]: pgmap v1833: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:28.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:28 smithi114 ceph-mon[42466]: pgmap v1833: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:29.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:28 smithi119 ceph-mon[33714]: pgmap v1833: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:30.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:30 smithi114 ceph-mon[42466]: pgmap v1834: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:30.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:30 smithi114 ceph-mon[30312]: pgmap v1834: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:31.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:30 smithi119 ceph-mon[33714]: pgmap v1834: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-24T12:15:32.855 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:32 smithi119 ceph-mon[33714]: pgmap v1835: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:15:32.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:32 smithi114 ceph-mon[30312]: pgmap v1835: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:15:32.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:32 smithi114 ceph-mon[42466]: pgmap v1835: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:15:33.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:15:33 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:15:34.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:34 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:34.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:34 smithi114 ceph-mon[30312]: pgmap v1836: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-24T12:15:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:34 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:34.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:34 smithi114 ceph-mon[42466]: pgmap v1836: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-24T12:15:34.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:15:34 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:15:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:35.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:34 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:35.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:34 smithi119 ceph-mon[33714]: pgmap v1836: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-24T12:15:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:15:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:36 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:36 smithi114 ceph-mon[30312]: pgmap v1837: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:15:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:36 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:36.738 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:36 smithi114 ceph-mon[42466]: pgmap v1837: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:15:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:36 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:36.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:36 smithi119 ceph-mon[33714]: pgmap v1837: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:38.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:38 smithi119 ceph-mon[33714]: pgmap v1838: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:38.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:38 smithi114 ceph-mon[30312]: pgmap v1838: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:38.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:38 smithi114 ceph-mon[42466]: pgmap v1838: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:40.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:40 smithi119 ceph-mon[33714]: pgmap v1839: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:40.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:40 smithi114 ceph-mon[30312]: pgmap v1839: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:40.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:40 smithi114 ceph-mon[42466]: pgmap v1839: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:42.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:42 smithi119 ceph-mon[33714]: pgmap v1840: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:42.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:42 smithi114 ceph-mon[30312]: pgmap v1840: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:42.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:42 smithi114 ceph-mon[42466]: pgmap v1840: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:43.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:15:43 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:15:44.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:44 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:44.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:44 smithi119 ceph-mon[33714]: pgmap v1841: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:44.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:44 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:44.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:44 smithi114 ceph-mon[30312]: pgmap v1841: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:44.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:15:44 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:15:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:44 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:44.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:44 smithi114 ceph-mon[42466]: pgmap v1841: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:46.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:46 smithi119 ceph-mon[33714]: pgmap v1842: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:46.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:46 smithi114 ceph-mon[30312]: pgmap v1842: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:46.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:46 smithi114 ceph-mon[42466]: pgmap v1842: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:48.623 INFO:tasks.workunit.client.0.smithi114.stderr:+ rados -p 5e219aa2-0962-4332-a6f6-f7f15fae1d00 put threemore /etc/passwd 2024-03-24T12:15:48.662 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 5e219aa2-0962-4332-a6f6-f7f15fae1d00 max_bytes 0 2024-03-24T12:15:48.753 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.749+0000 7f961c340640 1 -- 172.21.15.114:0/3123457404 >> v1:172.21.15.114:6790/0 conn(0x7f9614109fa0 legacy=0x7f961410c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:48.753 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- 172.21.15.114:0/3123457404 shutdown_connections 2024-03-24T12:15:48.753 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- 172.21.15.114:0/3123457404 >> 172.21.15.114:0/3123457404 conn(0x7f96140fb980 msgr2=0x7f96140fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- 172.21.15.114:0/3123457404 shutdown_connections 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- 172.21.15.114:0/3123457404 wait complete. 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 Processor -- start 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- start start 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96141a5e40 con 0x7f9614109fa0 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96141a7040 con 0x7f96141021a0 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f961c340640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96141a8240 con 0x7f961410dab0 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f96198db640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9614109fa0 0x7f96141a0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51694/0 (socket says 172.21.15.114:51694) 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.750+0000 7f96198db640 1 -- 172.21.15.114:0/3865950581 learned_addr learned my addr 172.21.15.114:0/3865950581 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499002060 0 0) 0x7f96141a5e40 con 0x7f9614109fa0 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9600004950 con 0x7f9614109fa0 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391541995 0 0) 0x7f96141a8240 con 0x7f961410dab0 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96141a5e40 con 0x7f961410dab0 2024-03-24T12:15:48.754 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497965421 0 0) 0x7f96141a7040 con 0x7f96141021a0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96141a8240 con 0x7f96141021a0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1272317290 0 0) 0x7f9600004950 con 0x7f9614109fa0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96141a7040 con 0x7f9614109fa0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9604004510 con 0x7f9614109fa0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 313691906 0 0) 0x7f96141a5e40 con 0x7f961410dab0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9600004950 con 0x7f961410dab0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f960c001bb0 con 0x7f961410dab0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3717548140 0 0) 0x7f96141a8240 con 0x7f96141021a0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96141a5e40 con 0x7f96141021a0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9610001800 con 0x7f96141021a0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2961946035 0 0) 0x7f96141a7040 con 0x7f9614109fa0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 >> v1:172.21.15.114:6790/0 conn(0x7f961410dab0 legacy=0x7f96141a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 >> v1:172.21.15.119:6789/0 conn(0x7f96141021a0 legacy=0x7f96141096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96141a9440 con 0x7f9614109fa0 2024-03-24T12:15:48.755 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96141a6030 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96141a64c0 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f9604003bf0 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.751+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f9604005180 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.752+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614069ef0 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.753+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f96040053c0 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.753+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(836..836 src has 254..836) v4 ==== 8190+0+0 (unknown 848865995 0 0) 0x7f9604088ea0 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.753+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=837}) v3 -- 0x7f96141a7040 con 0x7f9614109fa0 2024-03-24T12:15:48.756 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.758+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f9604055cd0 con 0x7f9614109fa0 2024-03-24T12:15:48.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:48 smithi119 ceph-mon[33714]: pgmap v1843: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:48.932 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:48.933+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f9614107c90 con 0x7f9614109fa0 2024-03-24T12:15:48.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:48 smithi114 ceph-mon[30312]: pgmap v1843: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:48.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:48 smithi114 ceph-mon[42466]: pgmap v1843: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:15:49.557 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:49.558+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v837) v1 ==== 217+0+0 (unknown 173465713 0 0) 0x7f9604054040 con 0x7f9614109fa0 2024-03-24T12:15:49.557 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:49.558+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 11 ==== osd_map(837..837 src has 254..837) v4 ==== 628+0+0 (unknown 277252697 0 0) 0x7f9604056590 con 0x7f9614109fa0 2024-03-24T12:15:49.557 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:49.558+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=838}) v3 -- 0x7f96141a8240 con 0x7f9614109fa0 2024-03-24T12:15:49.669 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:49.670+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f9614068260 con 0x7f9614109fa0 2024-03-24T12:15:49.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:49 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:15:49.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:49 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:15:50.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:49 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:15:50.559 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.559+0000 7f960b7fe640 1 -- 172.21.15.114:0/3865950581 <== mon.0 v1:172.21.15.114:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v838) v1 ==== 217+0+0 (unknown 2086109394 0 0) 0x7f9604055860 con 0x7f9614109fa0 2024-03-24T12:15:50.559 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_bytes = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:15:50.562 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.563+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 >> v1:172.21.15.114:6800/1253397276 conn(0x7f960006e500 legacy=0x7f96000709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:50.562 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.563+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 >> v1:172.21.15.114:6789/0 conn(0x7f9614109fa0 legacy=0x7f96141a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:50.562 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.563+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 shutdown_connections 2024-03-24T12:15:50.562 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.563+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 >> 172.21.15.114:0/3865950581 conn(0x7f96140fb980 msgr2=0x7f96140fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:15:50.563 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.563+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 shutdown_connections 2024-03-24T12:15:50.563 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.563+0000 7f961c340640 1 -- 172.21.15.114:0/3865950581 wait complete. 2024-03-24T12:15:50.576 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool set-quota 5e219aa2-0962-4332-a6f6-f7f15fae1d00 max_objects 0 2024-03-24T12:15:50.665 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.663+0000 7f2d7754b640 1 -- 172.21.15.114:0/2142082527 >> v1:172.21.15.114:6789/0 conn(0x7f2d7010c930 legacy=0x7f2d7010edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:50.665 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.663+0000 7f2d7754b640 1 -- 172.21.15.114:0/2142082527 shutdown_connections 2024-03-24T12:15:50.665 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.663+0000 7f2d7754b640 1 -- 172.21.15.114:0/2142082527 >> 172.21.15.114:0/2142082527 conn(0x7f2d700fb980 msgr2=0x7f2d700fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.663+0000 7f2d7754b640 1 -- 172.21.15.114:0/2142082527 shutdown_connections 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.663+0000 7f2d7754b640 1 -- 172.21.15.114:0/2142082527 wait complete. 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d7754b640 1 Processor -- start 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d7754b640 1 -- start start 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d7754b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2d701a5e40 con 0x7f2d7010c930 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d7754b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2d701a7040 con 0x7f2d70108dc0 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d7754b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2d701a8240 con 0x7f2d70100140 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d752e7640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2d70100140 0x7f2d70105380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53260/0 (socket says 172.21.15.114:53260) 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d752e7640 1 -- 172.21.15.114:0/1631930943 learned_addr learned my addr 172.21.15.114:0/1631930943 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269296576 0 0) 0x7f2d701a5e40 con 0x7f2d7010c930 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d5c001fe0 con 0x7f2d7010c930 2024-03-24T12:15:50.666 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578127899 0 0) 0x7f2d701a8240 con 0x7f2d70100140 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d701a5e40 con 0x7f2d70100140 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565785034 0 0) 0x7f2d701a7040 con 0x7f2d70108dc0 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.664+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d701a8240 con 0x7f2d70108dc0 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129026334 0 0) 0x7f2d5c001fe0 con 0x7f2d7010c930 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d701a7040 con 0x7f2d7010c930 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2d680017b0 con 0x7f2d7010c930 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4122853930 0 0) 0x7f2d701a5e40 con 0x7f2d70100140 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d5c001fe0 con 0x7f2d70100140 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2d6c001870 con 0x7f2d70100140 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1552254038 0 0) 0x7f2d701a8240 con 0x7f2d70108dc0 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d701a5e40 con 0x7f2d70108dc0 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2d600018d0 con 0x7f2d70108dc0 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3136022148 0 0) 0x7f2d701a7040 con 0x7f2d7010c930 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 >> v1:172.21.15.114:6790/0 conn(0x7f2d70100140 legacy=0x7f2d70105380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:50.667 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 >> v1:172.21.15.119:6789/0 conn(0x7f2d70108dc0 legacy=0x7f2d701a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:50.668 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d701a9440 con 0x7f2d7010c930 2024-03-24T12:15:50.668 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d701a6030 con 0x7f2d7010c930 2024-03-24T12:15:50.668 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d701a64c0 con 0x7f2d7010c930 2024-03-24T12:15:50.668 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f2d68004880 con 0x7f2d7010c930 2024-03-24T12:15:50.668 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.665+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f2d68004e80 con 0x7f2d7010c930 2024-03-24T12:15:50.668 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.666+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d54005070 con 0x7f2d7010c930 2024-03-24T12:15:50.668 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.666+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f2d68003970 con 0x7f2d7010c930 2024-03-24T12:15:50.669 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.667+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(838..838 src has 254..838) v4 ==== 8190+0+0 (unknown 1040308080 0 0) 0x7f2d68087cc0 con 0x7f2d7010c930 2024-03-24T12:15:50.669 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.667+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=839}) v3 -- 0x7f2d701a7040 con 0x7f2d7010c930 2024-03-24T12:15:50.670 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.671+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f2d68054a20 con 0x7f2d7010c930 2024-03-24T12:15:50.844 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:50.845+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f2d54005320 con 0x7f2d7010c930 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[30312]: pgmap v1844: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[30312]: osdmap e837: 8 total, 8 up, 8 in 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[42466]: pgmap v1844: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[42466]: osdmap e837: 8 total, 8 up, 8 in 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:15:50.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:50 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:51.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:50 smithi119 ceph-mon[33714]: pgmap v1844: 188 pgs: 188 active+clean; 859 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:15:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:15:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:50 smithi119 ceph-mon[33714]: osdmap e837: 8 total, 8 up, 8 in 2024-03-24T12:15:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:50 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-24T12:15:51.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:50 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:15:51.574 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:51.575+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v839) v1 ==== 221+0+0 (unknown 2715971441 0 0) 0x7f2d68052d90 con 0x7f2d7010c930 2024-03-24T12:15:51.575 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:51.576+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 11 ==== osd_map(839..839 src has 254..839) v4 ==== 628+0+0 (unknown 801991132 0 0) 0x7f2d68055350 con 0x7f2d7010c930 2024-03-24T12:15:51.575 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:51.576+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=840}) v3 -- 0x7f2d701a8240 con 0x7f2d7010c930 2024-03-24T12:15:51.685 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:51.686+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f2d54000c40 con 0x7f2d7010c930 2024-03-24T12:15:51.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:15:51.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:51 smithi114 ceph-mon[30312]: osdmap e838: 8 total, 8 up, 8 in 2024-03-24T12:15:51.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:51 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:15:51.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:15:51.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:51 smithi114 ceph-mon[42466]: osdmap e838: 8 total, 8 up, 8 in 2024-03-24T12:15:51.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:51 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:15:52.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/3865950581' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_bytes", "val": "0"}]': finished 2024-03-24T12:15:52.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:51 smithi119 ceph-mon[33714]: osdmap e838: 8 total, 8 up, 8 in 2024-03-24T12:15:52.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:51 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:15:52.581 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.581+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 v840) v1 ==== 221+0+0 (unknown 995333385 0 0) 0x7f2d680545b0 con 0x7f2d7010c930 2024-03-24T12:15:52.581 INFO:tasks.workunit.client.0.smithi114.stderr:set-quota max_objects = 0 for pool 5e219aa2-0962-4332-a6f6-f7f15fae1d00 2024-03-24T12:15:52.581 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.582+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 <== mon.0 v1:172.21.15.114:6789/0 13 ==== osd_map(840..840 src has 254..840) v4 ==== 628+0+0 (unknown 1620936069 0 0) 0x7f2d68058da0 con 0x7f2d7010c930 2024-03-24T12:15:52.581 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.582+0000 7f2d667fc640 1 -- 172.21.15.114:0/1631930943 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=841}) v3 -- 0x7f2d5c07c1c0 con 0x7f2d7010c930 2024-03-24T12:15:52.585 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.585+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 >> v1:172.21.15.114:6800/1253397276 conn(0x7f2d5c06e890 legacy=0x7f2d5c070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:52.585 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.585+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 >> v1:172.21.15.114:6789/0 conn(0x7f2d7010c930 legacy=0x7f2d701a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:15:52.585 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.585+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 shutdown_connections 2024-03-24T12:15:52.585 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.586+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 >> 172.21.15.114:0/1631930943 conn(0x7f2d700fb980 msgr2=0x7f2d7010fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:15:52.585 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.586+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 shutdown_connections 2024-03-24T12:15:52.585 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:15:52.586+0000 7f2d7754b640 1 -- 172.21.15.114:0/1631930943 wait complete. 2024-03-24T12:15:52.597 INFO:tasks.workunit.client.0.smithi114.stderr:+ sleep 30 2024-03-24T12:15:52.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[30312]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-24T12:15:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:15:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[30312]: osdmap e839: 8 total, 8 up, 8 in 2024-03-24T12:15:52.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:15:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[42466]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-24T12:15:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:15:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[42466]: osdmap e839: 8 total, 8 up, 8 in 2024-03-24T12:15:52.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:52 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:15:53.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:52 smithi119 ceph-mon[33714]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-24T12:15:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:15:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:52 smithi119 ceph-mon[33714]: osdmap e839: 8 total, 8 up, 8 in 2024-03-24T12:15:53.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:52 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]: dispatch 2024-03-24T12:15:53.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:15:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:15:53.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:53 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:15:53.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:53 smithi114 ceph-mon[30312]: osdmap e840: 8 total, 8 up, 8 in 2024-03-24T12:15:53.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:53 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:15:53.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:53 smithi114 ceph-mon[42466]: osdmap e840: 8 total, 8 up, 8 in 2024-03-24T12:15:54.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:53 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/1631930943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "field": "max_objects", "val": "0"}]': finished 2024-03-24T12:15:54.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:53 smithi119 ceph-mon[33714]: osdmap e840: 8 total, 8 up, 8 in 2024-03-24T12:15:54.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:54 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:54.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:54 smithi114 ceph-mon[42466]: pgmap v1850: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-24T12:15:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:54 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:54.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:54 smithi114 ceph-mon[30312]: pgmap v1850: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-24T12:15:54.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:15:54 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:15:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:15:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:54 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:15:55.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:54 smithi119 ceph-mon[33714]: pgmap v1850: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-24T12:15:56.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:56 smithi114 ceph-mon[30312]: pgmap v1851: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 863 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-24T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:56 smithi114 ceph-mon[42466]: pgmap v1851: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 863 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-24T12:15:57.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:56 smithi119 ceph-mon[33714]: pgmap v1851: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 863 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-24T12:15:58.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:15:58 smithi114 ceph-mon[30312]: pgmap v1852: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-24T12:15:58.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:15:58 smithi114 ceph-mon[42466]: pgmap v1852: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-24T12:15:59.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:15:58 smithi119 ceph-mon[33714]: pgmap v1852: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-24T12:16:00.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:00 smithi114 ceph-mon[30312]: pgmap v1853: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:16:00.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:00 smithi114 ceph-mon[42466]: pgmap v1853: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:16:01.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:00 smithi119 ceph-mon[33714]: pgmap v1853: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:16:02.855 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:02 smithi119 ceph-mon[33714]: pgmap v1854: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:02.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:02 smithi114 ceph-mon[30312]: pgmap v1854: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:02.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:02 smithi114 ceph-mon[42466]: pgmap v1854: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:16:03.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:16:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:16:04.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:04 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:04.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:04 smithi114 ceph-mon[30312]: pgmap v1855: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-24T12:16:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:04 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:04.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:04 smithi114 ceph-mon[42466]: pgmap v1855: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-24T12:16:04.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:04 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:16:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:04 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:05.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:04 smithi119 ceph-mon[33714]: pgmap v1855: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-24T12:16:05.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:05 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:05.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:05 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:06.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:05 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:06.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:06 smithi114 ceph-mon[30312]: pgmap v1856: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:06.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:06 smithi114 ceph-mon[42466]: pgmap v1856: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:07.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:06 smithi119 ceph-mon[33714]: pgmap v1856: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:08.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:08 smithi114 ceph-mon[30312]: pgmap v1857: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:08.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:08 smithi114 ceph-mon[42466]: pgmap v1857: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:09.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:08 smithi119 ceph-mon[33714]: pgmap v1857: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:10.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:10 smithi114 ceph-mon[30312]: pgmap v1858: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:10.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:10 smithi114 ceph-mon[42466]: pgmap v1858: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:11.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:10 smithi119 ceph-mon[33714]: pgmap v1858: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:12.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:12 smithi114 ceph-mon[42466]: pgmap v1859: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:12.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:12 smithi114 ceph-mon[30312]: pgmap v1859: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:13.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:12 smithi119 ceph-mon[33714]: pgmap v1859: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:13.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:16:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:16:14.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:14 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:14.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:14 smithi114 ceph-mon[42466]: pgmap v1860: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:14 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:14.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:14 smithi114 ceph-mon[30312]: pgmap v1860: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:14.988 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:14 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:16:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:15.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:14 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:15.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:14 smithi119 ceph-mon[33714]: pgmap v1860: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:15.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:15.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:15 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:16.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:16:16.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:16.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:16.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:16.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:15 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[30312]: pgmap v1861: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:16.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:16.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:16.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[42466]: pgmap v1861: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:16.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:16 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:16 smithi119 ceph-mon[33714]: pgmap v1861: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:16 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:16:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:16 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:16:17.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:16 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' 2024-03-24T12:16:18.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:18 smithi114 ceph-mon[30312]: pgmap v1862: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:18.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:18 smithi114 ceph-mon[42466]: pgmap v1862: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:19.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:18 smithi119 ceph-mon[33714]: pgmap v1862: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:20.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:20 smithi114 ceph-mon[42466]: pgmap v1863: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:20.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:20 smithi114 ceph-mon[42466]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:20 smithi114 ceph-mon[30312]: pgmap v1863: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:20.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:20 smithi114 ceph-mon[30312]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:20 smithi119 ceph-mon[33714]: pgmap v1863: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:21.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:20 smithi119 ceph-mon[33714]: from='mgr.14589 v1:172.21.15.114:0/91409870' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:16:22.598 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool delete 5e219aa2-0962-4332-a6f6-f7f15fae1d00 5e219aa2-0962-4332-a6f6-f7f15fae1d00 --yes-i-really-really-mean-it 2024-03-24T12:16:22.694 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.691+0000 7f764769d640 1 -- 172.21.15.114:0/3647207796 >> v1:172.21.15.114:6790/0 conn(0x7f764010da30 legacy=0x7f764010fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:22.694 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.691+0000 7f764769d640 1 -- 172.21.15.114:0/3647207796 shutdown_connections 2024-03-24T12:16:22.694 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.691+0000 7f764769d640 1 -- 172.21.15.114:0/3647207796 >> 172.21.15.114:0/3647207796 conn(0x7f76400751b0 msgr2=0x7f76400755c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:16:22.694 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.691+0000 7f764769d640 1 -- 172.21.15.114:0/3647207796 shutdown_connections 2024-03-24T12:16:22.694 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.691+0000 7f764769d640 1 -- 172.21.15.114:0/3647207796 wait complete. 2024-03-24T12:16:22.694 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.691+0000 7f764769d640 1 Processor -- start 2024-03-24T12:16:22.694 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f764769d640 1 -- start start 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f764769d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76401a5d80 con 0x7f7640109ec0 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f764769d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76401a6f80 con 0x7f76400ff9d0 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f764769d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76401a8180 con 0x7f764010da30 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f7644c38640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7640109ec0 0x7f76401a0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40500/0 (socket says 172.21.15.114:40500) 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f7644c38640 1 -- 172.21.15.114:0/469806399 learned_addr learned my addr 172.21.15.114:0/469806399 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f7645c3a640 1 --1- 172.21.15.114:0/469806399 >> v1:172.21.15.114:6790/0 conn(0x7f764010da30 0x7f76401a44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42010/0 (socket says 172.21.15.114:42010) 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934920582 0 0) 0x7f76401a8180 con 0x7f764010da30 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f762c003520 con 0x7f764010da30 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396269433 0 0) 0x7f76401a5d80 con 0x7f7640109ec0 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76401a8180 con 0x7f7640109ec0 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646078684 0 0) 0x7f76401a6f80 con 0x7f76400ff9d0 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76401a5d80 con 0x7f76400ff9d0 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224027961 0 0) 0x7f762c003520 con 0x7f764010da30 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76401a6f80 con 0x7f764010da30 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f76380017b0 con 0x7f764010da30 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862292142 0 0) 0x7f76401a8180 con 0x7f7640109ec0 2024-03-24T12:16:22.695 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f762c003520 con 0x7f7640109ec0 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f7630001850 con 0x7f7640109ec0 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007175651 0 0) 0x7f76401a5d80 con 0x7f76400ff9d0 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76401a8180 con 0x7f76400ff9d0 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f763c001830 con 0x7f76400ff9d0 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.692+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1771645416 0 0) 0x7f76401a6f80 con 0x7f764010da30 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.693+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 >> v1:172.21.15.119:6789/0 conn(0x7f76400ff9d0 legacy=0x7f7640106390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.693+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 >> v1:172.21.15.114:6789/0 conn(0x7f7640109ec0 legacy=0x7f76401a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.693+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76401a9380 con 0x7f764010da30 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.693+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76401a5f70 con 0x7f764010da30 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.693+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76401a6400 con 0x7f764010da30 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.693+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7f7638003e00 con 0x7f764010da30 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.693+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7f7638005110 con 0x7f764010da30 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.694+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7624005080 con 0x7f764010da30 2024-03-24T12:16:22.696 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.694+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7f7638003970 con 0x7f764010da30 2024-03-24T12:16:22.697 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.694+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(840..840 src has 254..840) v4 ==== 8190+0+0 (unknown 433039185 0 0) 0x7f7638088f70 con 0x7f764010da30 2024-03-24T12:16:22.697 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.695+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=841}) v3 -- 0x7f76401a6f80 con 0x7f764010da30 2024-03-24T12:16:22.699 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.699+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7f7638055d30 con 0x7f764010da30 2024-03-24T12:16:22.873 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:22.873+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7624005330 con 0x7f764010da30 2024-03-24T12:16:22.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:22 smithi114 ceph-mon[30312]: pgmap v1864: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:22.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:22 smithi114 ceph-mon[42466]: pgmap v1864: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:23.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:22 smithi119 ceph-mon[33714]: pgmap v1864: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:16:23.638 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:16:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug there is no tcmu-runner data available 2024-03-24T12:16:23.643 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.643+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 10 ==== osd_map(841..841 src has 254..841) v4 ==== 296+0+0 (unknown 1464362436 0 0) 0x7f7638003c20 con 0x7f764010da30 2024-03-24T12:16:23.643 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.643+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=842}) v3 -- 0x7f76401a5d80 con 0x7f764010da30 2024-03-24T12:16:23.646 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.647+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]=0 pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' removed v841) v1 ==== 248+0+0 (unknown 3753572194 0 0) 0x7f76380540a0 con 0x7f764010da30 2024-03-24T12:16:23.758 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.758+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7624001140 con 0x7f764010da30 2024-03-24T12:16:23.759 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.759+0000 7f76367fc640 1 -- 172.21.15.114:0/469806399 <== mon.2 v1:172.21.15.114:6790/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]=0 pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' does not exist v841) v1 ==== 255+0+0 (unknown 3158956147 0 0) 0x7f76380558c0 con 0x7f764010da30 2024-03-24T12:16:23.759 INFO:tasks.workunit.client.0.smithi114.stderr:pool '5e219aa2-0962-4332-a6f6-f7f15fae1d00' does not exist 2024-03-24T12:16:23.761 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.761+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 >> v1:172.21.15.114:6800/1253397276 conn(0x7f762c06e810 legacy=0x7f762c070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:23.761 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.761+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 >> v1:172.21.15.114:6790/0 conn(0x7f764010da30 legacy=0x7f76401a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:23.761 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.761+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 shutdown_connections 2024-03-24T12:16:23.761 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.761+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 >> 172.21.15.114:0/469806399 conn(0x7f76400751b0 msgr2=0x7f76401110e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:16:23.761 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.761+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 shutdown_connections 2024-03-24T12:16:23.761 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.761+0000 7f764769d640 1 -- 172.21.15.114:0/469806399 wait complete. 2024-03-24T12:16:23.770 INFO:tasks.workunit.client.0.smithi114.stderr:+ ceph osd pool delete 12c1aa96-773c-4a69-921d-e5e47a731694 12c1aa96-773c-4a69-921d-e5e47a731694 --yes-i-really-really-mean-it 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.852+0000 7fd83d80e640 1 -- 172.21.15.114:0/3604882829 >> v1:172.21.15.114:6790/0 conn(0x7fd838109eb0 legacy=0x7fd83810c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.852+0000 7fd83d80e640 1 -- 172.21.15.114:0/3604882829 shutdown_connections 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.852+0000 7fd83d80e640 1 -- 172.21.15.114:0/3604882829 >> 172.21.15.114:0/3604882829 conn(0x7fd8380fca70 msgr2=0x7fd8380fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.852+0000 7fd83d80e640 1 -- 172.21.15.114:0/3604882829 shutdown_connections 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.852+0000 7fd83d80e640 1 -- 172.21.15.114:0/3604882829 wait complete. 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83d80e640 1 Processor -- start 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83d80e640 1 -- start start 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83d80e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8381a5e60 con 0x7fd838109eb0 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83d80e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8381a7060 con 0x7fd838101230 2024-03-24T12:16:23.855 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83d80e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8381a8260 con 0x7fd83810da20 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd8377fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd83810da20 0x7fd8381a4560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42038/0 (socket says 172.21.15.114:42038) 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd8377fe640 1 -- 172.21.15.114:0/2609347058 learned_addr learned my addr 172.21.15.114:0/2609347058 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd8367fc640 1 --1- 172.21.15.114:0/2609347058 >> v1:172.21.15.114:6789/0 conn(0x7fd838109eb0 0x7fd8381a0d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40526/0 (socket says 172.21.15.114:40526) 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001857558 0 0) 0x7fd8381a8260 con 0x7fd83810da20 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd820003520 con 0x7fd83810da20 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634646239 0 0) 0x7fd8381a5e60 con 0x7fd838109eb0 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8381a8260 con 0x7fd838109eb0 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10920942 0 0) 0x7fd8381a7060 con 0x7fd838101230 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8381a5e60 con 0x7fd838101230 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1238260019 0 0) 0x7fd820003520 con 0x7fd83810da20 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8381a7060 con 0x7fd83810da20 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.853+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fd830001d70 con 0x7fd83810da20 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540656182 0 0) 0x7fd8381a8260 con 0x7fd838109eb0 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd820003520 con 0x7fd838109eb0 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fd82c004460 con 0x7fd838109eb0 2024-03-24T12:16:23.856 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 830121420 0 0) 0x7fd8381a7060 con 0x7fd83810da20 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 >> v1:172.21.15.119:6789/0 conn(0x7fd838101230 legacy=0x7fd838106470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 >> v1:172.21.15.114:6789/0 conn(0x7fd838109eb0 legacy=0x7fd8381a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8381a9460 con 0x7fd83810da20 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8381a6050 con 0x7fd83810da20 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8381a64e0 con 0x7fd83810da20 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2359058675 0 0) 0x7fd830003e00 con 0x7fd83810da20 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.854+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2014693838 0 0) 0x7fd830005e80 con 0x7fd83810da20 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.855+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd824005080 con 0x7fd83810da20 2024-03-24T12:16:23.857 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.855+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 21) v1 ==== 91104+0+0 (unknown 3107866384 0 0) 0x7fd830003970 con 0x7fd83810da20 2024-03-24T12:16:23.858 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.856+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(841..841 src has 254..841) v4 ==== 7802+0+0 (unknown 4194737482 0 0) 0x7fd830088cd0 con 0x7fd83810da20 2024-03-24T12:16:23.858 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.856+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=842}) v3 -- 0x7fd8381a7060 con 0x7fd83810da20 2024-03-24T12:16:23.860 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:23.860+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (unknown 1092875540 0 1883366685) 0x7fd830055cf0 con 0x7fd83810da20 2024-03-24T12:16:23.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:23 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/469806399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:23.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:23 smithi114 ceph-mon[42466]: from='client.56534 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:23 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/469806399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:23.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:23 smithi114 ceph-mon[30312]: from='client.56534 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.034 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.033+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd824005330 con 0x7fd83810da20 2024-03-24T12:16:24.052 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:23 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/469806399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:23 smithi119 ceph-mon[33714]: from='client.56534 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.657 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.657+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 10 ==== osd_map(842..842 src has 254..842) v4 ==== 296+0+0 (unknown 1269011044 0 0) 0x7fd830056620 con 0x7fd83810da20 2024-03-24T12:16:24.657 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.657+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=843}) v3 -- 0x7fd8381a5e60 con 0x7fd83810da20 2024-03-24T12:16:24.659 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.659+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]=0 pool '12c1aa96-773c-4a69-921d-e5e47a731694' removed v842) v1 ==== 248+0+0 (unknown 2849625696 0 0) 0x7fd830054060 con 0x7fd83810da20 2024-03-24T12:16:24.776 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.776+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd824001140 con 0x7fd83810da20 2024-03-24T12:16:24.776 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.777+0000 7fd83c80c640 1 -- 172.21.15.114:0/2609347058 <== mon.2 v1:172.21.15.114:6790/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]=0 pool '12c1aa96-773c-4a69-921d-e5e47a731694' does not exist v842) v1 ==== 255+0+0 (unknown 3266540153 0 0) 0x7fd830055880 con 0x7fd83810da20 2024-03-24T12:16:24.776 INFO:tasks.workunit.client.0.smithi114.stderr:pool '12c1aa96-773c-4a69-921d-e5e47a731694' does not exist 2024-03-24T12:16:24.779 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.778+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 >> v1:172.21.15.114:6800/1253397276 conn(0x7fd82006e2e0 legacy=0x7fd8200707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:24.779 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.778+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 >> v1:172.21.15.114:6790/0 conn(0x7fd83810da20 legacy=0x7fd8381a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T12:16:24.779 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.779+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 shutdown_connections 2024-03-24T12:16:24.779 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.779+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 >> 172.21.15.114:0/2609347058 conn(0x7fd8380fca70 msgr2=0x7fd8381106b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T12:16:24.779 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.779+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 shutdown_connections 2024-03-24T12:16:24.779 INFO:tasks.workunit.client.0.smithi114.stderr:2024-03-24T12:16:24.779+0000 7fd83d80e640 1 -- 172.21.15.114:0/2609347058 wait complete. 2024-03-24T12:16:24.787 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo OK 2024-03-24T12:16:24.788 INFO:tasks.workunit.client.0.smithi114.stdout:OK 2024-03-24T12:16:24.789 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-24T12:16:24.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-24T12:16:24.858 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-24T12:16:24.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: ::ffff:172.21.15.119 - - [24/Mar/2024:12:16:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: pgmap v1865: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: from='client.56534 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: osdmap e841: 8 total, 8 up, 8 in 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/469806399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: from='client.56534 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2609347058' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[30312]: from='client.56540 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: pgmap v1865: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: from='client.56534 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: osdmap e841: 8 total, 8 up, 8 in 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/469806399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: from='client.56534 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2609347058' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:24.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:24 smithi114 ceph-mon[42466]: from='client.56540 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: from='client.24448 v1:172.21.15.119:0/2362955723' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:16:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: pgmap v1865: 188 pgs: 188 active+clean; 871 KiB data, 911 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:16:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: from='client.56534 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:16:25.053 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: osdmap e841: 8 total, 8 up, 8 in 2024-03-24T12:16:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/469806399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: from='client.56534 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "pool2": "5e219aa2-0962-4332-a6f6-f7f15fae1d00", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2609347058' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:24 smithi119 ceph-mon[33714]: from='client.56540 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.182 DEBUG:teuthology.parallel:result is None 2024-03-24T12:16:25.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T12:16:25.249 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T12:16:25.249 DEBUG:teuthology.orchestra.run.smithi114:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-24T12:16:25.317 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T12:16:25.317 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T12:16:25.327 INFO:tasks.cephadm:Teardown begin 2024-03-24T12:16:25.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:16:25.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:16:25.419 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T12:16:25.419 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:16:25.439 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:16:25.475 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T12:16:25.476 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T12:16:25.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a 2024-03-24T12:16:25.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[42466]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:16:25.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[42466]: from='client.56540 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:16:25.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[42466]: osdmap e842: 8 total, 8 up, 8 in 2024-03-24T12:16:25.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[42466]: from='client.? v1:172.21.15.114:0/2609347058' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[42466]: from='client.56540 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 systemd[1]: Stopping Ceph mon.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:25.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[30312]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:16:25.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[30312]: from='client.56540 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:16:25.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[30312]: osdmap e842: 8 total, 8 up, 8 in 2024-03-24T12:16:25.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[30312]: from='client.? v1:172.21.15.114:0/2609347058' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-mon[30312]: from='client.56540 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:25 smithi119 ceph-mon[33714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-24T12:16:25.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:25 smithi119 ceph-mon[33714]: from='client.56540 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:16:25.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:25 smithi119 ceph-mon[33714]: osdmap e842: 8 total, 8 up, 8 in 2024-03-24T12:16:25.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:25 smithi119 ceph-mon[33714]: from='client.? v1:172.21.15.114:0/2609347058' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:25.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:25 smithi119 ceph-mon[33714]: from='client.56540 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "12c1aa96-773c-4a69-921d-e5e47a731694", "pool2": "12c1aa96-773c-4a69-921d-e5e47a731694", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:16:26.164 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a[30289]: 2024-03-24T12:16:25.765+0000 7f0515c21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:26.164 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a[30289]: 2024-03-24T12:16:25.765+0000 7f0515c21700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T12:16:26.164 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:25 smithi114 podman[160700]: 2024-03-24 12:16:25.889519395 +0000 UTC m=+0.225871373 container died a9e456820c15a3664054c80dea0a630340377bb17179435f2ebaaf2d489b9b63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T12:16:26.164 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:25] ENGINE Bus STOPPING 2024-03-24T12:16:26.165 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:16:26.165 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:25] ENGINE Bus STOPPED 2024-03-24T12:16:26.165 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:25] ENGINE Bus STARTING 2024-03-24T12:16:26.165 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:25] ENGINE Serving on http://:::9283 2024-03-24T12:16:26.165 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:25 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:25] ENGINE Bus STARTED 2024-03-24T12:16:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:26 smithi114 podman[160700]: 2024-03-24 12:16:26.339941641 +0000 UTC m=+0.676293665 container cleanup a9e456820c15a3664054c80dea0a630340377bb17179435f2ebaaf2d489b9b63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T12:16:26.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Mar 24 12:16:26 smithi114 bash[160700]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-a 2024-03-24T12:16:26.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T12:16:26.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:26.801 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T12:16:26.801 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-24T12:16:26.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.c 2024-03-24T12:16:27.129 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:26 smithi114 systemd[1]: Stopping Ceph mon.c for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:27.460 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-c[42443]: 2024-03-24T12:16:27.129+0000 7fd0e0027700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:27.460 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-c[42443]: 2024-03-24T12:16:27.129+0000 7fd0e0027700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T12:16:27.460 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:27 smithi114 podman[160843]: 2024-03-24 12:16:27.181997933 +0000 UTC m=+0.156405618 container died 6270eb72bb93612557d4b3c6cf3eb2e34df746d87d0e0a2625ab1b4fd3ac636d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-c, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T12:16:27.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:27] ENGINE Bus STOPPING 2024-03-24T12:16:27.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:16:27.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:27] ENGINE Bus STOPPED 2024-03-24T12:16:27.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:27] ENGINE Bus STARTING 2024-03-24T12:16:27.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:27] ENGINE Serving on http://:::9283 2024-03-24T12:16:27.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:27 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: [24/Mar/2024:12:16:27] ENGINE Bus STARTED 2024-03-24T12:16:27.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:27 smithi114 podman[160843]: 2024-03-24 12:16:27.569590879 +0000 UTC m=+0.543998563 container cleanup 6270eb72bb93612557d4b3c6cf3eb2e34df746d87d0e0a2625ab1b4fd3ac636d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T12:16:27.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Mar 24 12:16:27 smithi114 bash[160843]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-c 2024-03-24T12:16:27.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T12:16:27.996 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:27.997 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-24T12:16:27.997 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-24T12:16:27.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.b 2024-03-24T12:16:28.303 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:28 smithi119 systemd[1]: Stopping Ceph mon.b for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:28.802 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:28 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b[33691]: 2024-03-24T12:16:28.416+0000 7fc093208700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:28 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b[33691]: 2024-03-24T12:16:28.416+0000 7fc093208700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T12:16:28.803 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:28 smithi119 podman[79073]: 2024-03-24 12:16:28.570728309 +0000 UTC m=+0.376944091 container died fd6a85ca5007277daf365e6cefd9bf947757612e9d94ffc9628990cf94837518 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:16:29.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:29 smithi119 podman[79073]: 2024-03-24 12:16:29.013799772 +0000 UTC m=+0.820015554 container cleanup fd6a85ca5007277daf365e6cefd9bf947757612e9d94ffc9628990cf94837518 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:16:29.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 24 12:16:29 smithi119 bash[79073]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mon-b 2024-03-24T12:16:29.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T12:16:29.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:29.474 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-24T12:16:29.474 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T12:16:29.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.y 2024-03-24T12:16:29.758 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:29 smithi114 systemd[1]: Stopping Ceph mgr.y for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:30.065 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T12:16:29.759+0000 7f385cd0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:30.065 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:29 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y[30570]: 2024-03-24T12:16:29.759+0000 7f385cd0b700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T12:16:30.065 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:29 smithi114 podman[160985]: 2024-03-24 12:16:29.791224275 +0000 UTC m=+0.126382789 container died f3074bd1a81c5e973a15fd36a98e2b34d298b155d424b5287d0221f714cbe98c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-24T12:16:30.407 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:30 smithi114 podman[160985]: 2024-03-24 12:16:30.165969789 +0000 UTC m=+0.501128292 container cleanup f3074bd1a81c5e973a15fd36a98e2b34d298b155d424b5287d0221f714cbe98c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T12:16:30.407 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 24 12:16:30 smithi114 bash[160985]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-y 2024-03-24T12:16:30.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T12:16:30.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:30.564 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T12:16:30.564 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T12:16:30.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.x 2024-03-24T12:16:30.875 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:30 smithi119 systemd[1]: Stopping Ceph mgr.x for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:31.126 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:30 smithi119 podman[79202]: 2024-03-24 12:16:30.877023142 +0000 UTC m=+0.127753925 container died f224daaff314a729dd72599eb51c79e5d7cc2962cb63f5c85a8a83d0696c0fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T12:16:31.426 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:31 smithi119 podman[79202]: 2024-03-24 12:16:31.201596616 +0000 UTC m=+0.452327396 container cleanup f224daaff314a729dd72599eb51c79e5d7cc2962cb63f5c85a8a83d0696c0fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T12:16:31.427 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:31 smithi119 bash[79202]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x 2024-03-24T12:16:31.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T12:16:31.687 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:31 smithi119 podman[79221]: 2024-03-24 12:16:31.428115146 +0000 UTC m=+0.556013769 container remove f224daaff314a729dd72599eb51c79e5d7cc2962cb63f5c85a8a83d0696c0fbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-mgr-x, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T12:16:31.687 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:31 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:16:31.687 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:31 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-24T12:16:31.687 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:31 smithi119 systemd[1]: Stopped Ceph mgr.x for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:16:31.687 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 24 12:16:31 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@mgr.x.service: Consumed 34.853s CPU time. 2024-03-24T12:16:31.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:31.701 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T12:16:31.701 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T12:16:31.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.0 2024-03-24T12:16:31.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:31 smithi114 systemd[1]: Stopping Ceph osd.0 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:32.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:31 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0[46853]: 2024-03-24T12:16:31.988+0000 7f2fcc8fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:32.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:31 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0[46853]: 2024-03-24T12:16:31.988+0000 7f2fcc8fe700 -1 osd.0 842 *** Got signal Terminated *** 2024-03-24T12:16:32.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:31 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0[46853]: 2024-03-24T12:16:31.988+0000 7f2fcc8fe700 -1 osd.0 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:16:37.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:37 smithi114 podman[161118]: 2024-03-24 12:16:37.309959755 +0000 UTC m=+5.419411770 container died 6fe55b4fce4b6825e455c416d79a1c02ebd245a043e1eab9a788458d28f51059 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:16:37.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:37 smithi114 podman[161118]: 2024-03-24 12:16:37.703131891 +0000 UTC m=+5.812583905 container cleanup 6fe55b4fce4b6825e455c416d79a1c02ebd245a043e1eab9a788458d28f51059 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-24T12:16:37.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:37 smithi114 bash[161118]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0 2024-03-24T12:16:38.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:37 smithi114 podman[161152]: 2024-03-24 12:16:37.929318619 +0000 UTC m=+0.609722974 container remove 6fe55b4fce4b6825e455c416d79a1c02ebd245a043e1eab9a788458d28f51059 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T12:16:38.556 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:38 smithi114 podman[161248]: 2024-03-24 12:16:38.160985523 +0000 UTC m=+0.017419092 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:16:38.556 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:38 smithi114 podman[161248]: 2024-03-24 12:16:38.263459285 +0000 UTC m=+0.119892852 container create 2361d8aa059e0cd10de6fde512507c752df6c118d721cdaaf26ba636dcb1a1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T12:16:38.556 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:38 smithi114 podman[161248]: 2024-03-24 12:16:38.476228269 +0000 UTC m=+0.332661849 container init 2361d8aa059e0cd10de6fde512507c752df6c118d721cdaaf26ba636dcb1a1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:16:38.556 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:38 smithi114 podman[161248]: 2024-03-24 12:16:38.482287214 +0000 UTC m=+0.338720799 container start 2361d8aa059e0cd10de6fde512507c752df6c118d721cdaaf26ba636dcb1a1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T12:16:38.556 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:38 smithi114 podman[161248]: 2024-03-24 12:16:38.557596538 +0000 UTC m=+0.414030130 container attach 2361d8aa059e0cd10de6fde512507c752df6c118d721cdaaf26ba636dcb1a1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T12:16:38.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:38 smithi114 podman[161248]: 2024-03-24 12:16:38.77297739 +0000 UTC m=+0.629410957 container died 2361d8aa059e0cd10de6fde512507c752df6c118d721cdaaf26ba636dcb1a1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T12:16:39.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T12:16:39.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:39 smithi114 podman[161294]: 2024-03-24 12:16:39.327651561 +0000 UTC m=+0.549897968 container remove 2361d8aa059e0cd10de6fde512507c752df6c118d721cdaaf26ba636dcb1a1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T12:16:39.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:39 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T12:16:39.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:39 smithi114 systemd[1]: Stopped Ceph osd.0 for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:16:39.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Mar 24 12:16:39 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.0.service: Consumed 59.415s CPU time. 2024-03-24T12:16:39.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:39.393 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T12:16:39.393 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T12:16:39.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.1 2024-03-24T12:16:39.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:39 smithi114 systemd[1]: Stopping Ceph osd.1 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:39.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:39 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1[52784]: 2024-03-24T12:16:39.699+0000 7f0adcff6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:39.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:39 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1[52784]: 2024-03-24T12:16:39.699+0000 7f0adcff6700 -1 osd.1 842 *** Got signal Terminated *** 2024-03-24T12:16:39.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:39 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1[52784]: 2024-03-24T12:16:39.699+0000 7f0adcff6700 -1 osd.1 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:16:45.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:45 smithi114 podman[161379]: 2024-03-24 12:16:45.262920344 +0000 UTC m=+5.648988665 container died 08bcea217db2065fcb6567594485fe7230f1119a0b53de03fa9e455d1e062654 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T12:16:45.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:45 smithi114 podman[161379]: 2024-03-24 12:16:45.655833802 +0000 UTC m=+6.041902222 container cleanup 08bcea217db2065fcb6567594485fe7230f1119a0b53de03fa9e455d1e062654 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True, GIT_CLEAN=True) 2024-03-24T12:16:45.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:45 smithi114 bash[161379]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1 2024-03-24T12:16:46.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:45 smithi114 podman[161403]: 2024-03-24 12:16:45.873578455 +0000 UTC m=+0.607109778 container remove 08bcea217db2065fcb6567594485fe7230f1119a0b53de03fa9e455d1e062654 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T12:16:46.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:46 smithi114 podman[161503]: 2024-03-24 12:16:46.084677866 +0000 UTC m=+0.018259172 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:16:46.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:46 smithi114 podman[161503]: 2024-03-24 12:16:46.1994171 +0000 UTC m=+0.132998407 container create c381fd77cf06aa4c0672ef1cb29cb3e3fac00470a6d30193207f5d673ea91996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T12:16:46.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:46 smithi114 podman[161503]: 2024-03-24 12:16:46.426813108 +0000 UTC m=+0.360394430 container init c381fd77cf06aa4c0672ef1cb29cb3e3fac00470a6d30193207f5d673ea91996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-24T12:16:46.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:46 smithi114 podman[161503]: 2024-03-24 12:16:46.432887261 +0000 UTC m=+0.366468578 container start c381fd77cf06aa4c0672ef1cb29cb3e3fac00470a6d30193207f5d673ea91996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T12:16:46.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:46 smithi114 podman[161503]: 2024-03-24 12:16:46.526742056 +0000 UTC m=+0.460323375 container attach c381fd77cf06aa4c0672ef1cb29cb3e3fac00470a6d30193207f5d673ea91996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T12:16:46.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:46 smithi114 podman[161503]: 2024-03-24 12:16:46.72711056 +0000 UTC m=+0.660691867 container died c381fd77cf06aa4c0672ef1cb29cb3e3fac00470a6d30193207f5d673ea91996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T12:16:47.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T12:16:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:47 smithi114 podman[161549]: 2024-03-24 12:16:47.221874928 +0000 UTC m=+0.489639237 container remove c381fd77cf06aa4c0672ef1cb29cb3e3fac00470a6d30193207f5d673ea91996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T12:16:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:47 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-24T12:16:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:47 smithi114 systemd[1]: Stopped Ceph osd.1 for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:16:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 24 12:16:47 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.1.service: Consumed 1min 19ms CPU time. 2024-03-24T12:16:47.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:47.285 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T12:16:47.285 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T12:16:47.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.2 2024-03-24T12:16:47.601 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:47 smithi114 systemd[1]: Stopping Ceph osd.2 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:47.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:47 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2[58901]: 2024-03-24T12:16:47.602+0000 7f137f1fc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:47.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:47 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2[58901]: 2024-03-24T12:16:47.602+0000 7f137f1fc700 -1 osd.2 842 *** Got signal Terminated *** 2024-03-24T12:16:47.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:47 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2[58901]: 2024-03-24T12:16:47.602+0000 7f137f1fc700 -1 osd.2 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:16:53.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:52 smithi114 podman[161632]: 2024-03-24 12:16:52.939253878 +0000 UTC m=+5.433387999 container died e0bb6a9a4c7de19481bf67a7518c3b17157049e6e5d8a39cfb2d52f7baf9daf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:16:53.565 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:53 smithi114 podman[161632]: 2024-03-24 12:16:53.332966206 +0000 UTC m=+5.827100327 container cleanup e0bb6a9a4c7de19481bf67a7518c3b17157049e6e5d8a39cfb2d52f7baf9daf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T12:16:53.566 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:53 smithi114 bash[161632]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2 2024-03-24T12:16:53.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:53 smithi114 podman[161654]: 2024-03-24 12:16:53.567265611 +0000 UTC m=+0.620047556 container remove e0bb6a9a4c7de19481bf67a7518c3b17157049e6e5d8a39cfb2d52f7baf9daf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:16:54.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:53 smithi114 podman[161747]: 2024-03-24 12:16:53.893078272 +0000 UTC m=+0.114081038 container create f46493d98aecd6f68ef49ddcb9550f3ebd77174aac155dfdb1fafe461b33182d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:16:54.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:53 smithi114 podman[161747]: 2024-03-24 12:16:53.796660367 +0000 UTC m=+0.017663155 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:16:54.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:54 smithi114 podman[161747]: 2024-03-24 12:16:54.095237042 +0000 UTC m=+0.316239826 container init f46493d98aecd6f68ef49ddcb9550f3ebd77174aac155dfdb1fafe461b33182d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True) 2024-03-24T12:16:54.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:54 smithi114 podman[161747]: 2024-03-24 12:16:54.101117797 +0000 UTC m=+0.322120566 container start f46493d98aecd6f68ef49ddcb9550f3ebd77174aac155dfdb1fafe461b33182d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:16:54.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:54 smithi114 podman[161747]: 2024-03-24 12:16:54.178411635 +0000 UTC m=+0.399414412 container attach f46493d98aecd6f68ef49ddcb9550f3ebd77174aac155dfdb1fafe461b33182d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:16:54.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:54 smithi114 podman[161747]: 2024-03-24 12:16:54.386864151 +0000 UTC m=+0.607866917 container died f46493d98aecd6f68ef49ddcb9550f3ebd77174aac155dfdb1fafe461b33182d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:16:55.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T12:16:55.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:54 smithi114 podman[161793]: 2024-03-24 12:16:54.982030688 +0000 UTC m=+0.590238863 container remove f46493d98aecd6f68ef49ddcb9550f3ebd77174aac155dfdb1fafe461b33182d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T12:16:55.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:55 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-24T12:16:55.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:55 smithi114 systemd[1]: Stopped Ceph osd.2 for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:16:55.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Mar 24 12:16:55 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.2.service: Consumed 54.152s CPU time. 2024-03-24T12:16:55.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:16:55.046 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T12:16:55.046 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T12:16:55.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.3 2024-03-24T12:16:55.394 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:16:55 smithi114 systemd[1]: Stopping Ceph osd.3 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:16:55.737 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:16:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3[65524]: 2024-03-24T12:16:55.395+0000 7fdb60e94700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:16:55.737 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:16:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3[65524]: 2024-03-24T12:16:55.395+0000 7fdb60e94700 -1 osd.3 842 *** Got signal Terminated *** 2024-03-24T12:16:55.737 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:16:55 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3[65524]: 2024-03-24T12:16:55.395+0000 7fdb60e94700 -1 osd.3 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:16:59.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:16:58 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:16:58.622+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:16:59.953 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:16:59 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:16:59.629+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:00.302 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:16:59 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:16:59.954+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:00.947 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:00.674+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:01.099 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:00 smithi114 podman[161877]: 2024-03-24 12:17:00.811353324 +0000 UTC m=+5.543290788 container died bc52725e169d08913f60ad1a3c32005dc59f24162a6ccab523a04e73a5f8a07f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T12:17:01.302 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:00 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:00.948+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:01.487 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 podman[161877]: 2024-03-24 12:17:01.101031298 +0000 UTC m=+5.832968764 container cleanup bc52725e169d08913f60ad1a3c32005dc59f24162a6ccab523a04e73a5f8a07f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T12:17:01.487 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 bash[161877]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3 2024-03-24T12:17:01.488 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 podman[161899]: 2024-03-24 12:17:01.343597242 +0000 UTC m=+0.530065547 container remove bc52725e169d08913f60ad1a3c32005dc59f24162a6ccab523a04e73a5f8a07f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-24T12:17:01.634 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:01.340+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.082988+0000 front 2024-03-24T12:16:35.082998+0000 (oldest deadline 2024-03-24T12:17:00.982857+0000) 2024-03-24T12:17:01.634 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:01.467+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:01.938 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:01.634+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:01.970 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 podman[161997]: 2024-03-24 12:17:01.564111613 +0000 UTC m=+0.017148117 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:01.970 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 podman[161997]: 2024-03-24 12:17:01.677785122 +0000 UTC m=+0.130821698 container create 102e29b144cfb001890fd898a8aad25d86ce80f8067494770eed690a60971378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:01.970 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 podman[161997]: 2024-03-24 12:17:01.888777156 +0000 UTC m=+0.341813656 container init 102e29b144cfb001890fd898a8aad25d86ce80f8067494770eed690a60971378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:01.970 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 podman[161997]: 2024-03-24 12:17:01.894594957 +0000 UTC m=+0.347631460 container start 102e29b144cfb001890fd898a8aad25d86ce80f8067494770eed690a60971378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T12:17:01.970 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:01 smithi114 podman[161997]: 2024-03-24 12:17:01.971730789 +0000 UTC m=+0.424767303 container attach 102e29b144cfb001890fd898a8aad25d86ce80f8067494770eed690a60971378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T12:17:02.237 INFO:journalctl@ceph.osd.3.smithi114.stdout:Mar 24 12:17:02 smithi114 podman[161997]: 2024-03-24 12:17:02.184608188 +0000 UTC m=+0.637644697 container died 102e29b144cfb001890fd898a8aad25d86ce80f8067494770eed690a60971378 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T12:17:02.302 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:01 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:01.939+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:02.574 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:02.381+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.082988+0000 front 2024-03-24T12:16:35.082998+0000 (oldest deadline 2024-03-24T12:17:00.982857+0000) 2024-03-24T12:17:02.574 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:02.464+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:02.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T12:17:02.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:02.795 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T12:17:02.795 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T12:17:02.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.4 2024-03-24T12:17:02.837 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:02.651+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:03.087 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:02 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:02.979+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:03.088 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:02 smithi119 systemd[1]: Stopping Ceph osd.4 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:03.088 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:03.088+0000 7f40bc836700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:03.088 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:03.088+0000 7f40bc836700 -1 osd.4 842 *** Got signal Terminated *** 2024-03-24T12:17:03.088 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:03.088+0000 7f40bc836700 -1 osd.4 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:03.677 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:03.447+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:03.677 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:03.395+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.082988+0000 front 2024-03-24T12:16:35.082998+0000 (oldest deadline 2024-03-24T12:17:00.982857+0000) 2024-03-24T12:17:03.994 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:03.677+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:04.302 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:03 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:03.995+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:04.803 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:04 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:04.403+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:04.803 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:04 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:04.645+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:04.803 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:04 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:04.405+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.082988+0000 front 2024-03-24T12:16:35.082998+0000 (oldest deadline 2024-03-24T12:17:00.982857+0000) 2024-03-24T12:17:05.302 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:04 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:04.949+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:05.802 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:05 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:05.614+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:05.803 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:05 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:05.395+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:05.803 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:05 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:05.440+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.082988+0000 front 2024-03-24T12:16:35.082998+0000 (oldest deadline 2024-03-24T12:17:00.982857+0000) 2024-03-24T12:17:06.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:05 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:05.900+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:06.663 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:06 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:06.432+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:06.663 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:06 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:06.432+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:06.664 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:06 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:06.402+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.082988+0000 front 2024-03-24T12:16:35.082998+0000 (oldest deadline 2024-03-24T12:17:00.982857+0000) 2024-03-24T12:17:06.664 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:06 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:06.402+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:41.483712+0000 front 2024-03-24T12:16:41.483826+0000 (oldest deadline 2024-03-24T12:17:06.183691+0000) 2024-03-24T12:17:06.937 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:06 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:06.664+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:06.938 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:06 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:06.664+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:07.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:06 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:06.938+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:07.713 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:07.386+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.082988+0000 front 2024-03-24T12:16:35.082998+0000 (oldest deadline 2024-03-24T12:17:00.982857+0000) 2024-03-24T12:17:07.713 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4[39235]: 2024-03-24T12:17:07.386+0000 7f40b8e2a700 -1 osd.4 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:41.483712+0000 front 2024-03-24T12:16:41.483826+0000 (oldest deadline 2024-03-24T12:17:06.183691+0000) 2024-03-24T12:17:07.713 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:07.393+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:07.713 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:07.393+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:08.053 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:07.910+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:08.053 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:07.910+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:08.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:07.713+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:08.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:07 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:07.713+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:08.642 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:08 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:08.363+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:08.642 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:08 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:08.363+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:08.894 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:08 smithi119 podman[79359]: 2024-03-24 12:17:08.643300301 +0000 UTC m=+5.661374241 container died f8cc3ddd62f76a1c6a0a88dd2c2572ad2f39811e7116e6104e44f864ec9c1d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T12:17:08.895 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:08 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:08.698+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:08.895 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:08 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:08.698+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:09.161 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:08 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:08.907+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:09.161 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:08 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:08.908+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:09.161 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:08 smithi119 podman[79359]: 2024-03-24 12:17:08.986195906 +0000 UTC m=+6.004269916 container cleanup f8cc3ddd62f76a1c6a0a88dd2c2572ad2f39811e7116e6104e44f864ec9c1d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T12:17:09.161 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:08 smithi119 bash[79359]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4 2024-03-24T12:17:09.552 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:09 smithi119 podman[79386]: 2024-03-24 12:17:09.162252429 +0000 UTC m=+0.516042655 container remove f8cc3ddd62f76a1c6a0a88dd2c2572ad2f39811e7116e6104e44f864ec9c1d49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T12:17:09.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:09 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:09.323+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:09.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:09 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:09.323+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:09.922 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:09 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:09.654+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:09.923 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:09 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:09.654+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:09.923 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:09 smithi119 podman[79483]: 2024-03-24 12:17:09.506995583 +0000 UTC m=+0.017607128 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:09.923 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:09 smithi119 podman[79483]: 2024-03-24 12:17:09.621428829 +0000 UTC m=+0.132040381 container create 4217b809fbcab50cb8e67b0758fd9cb99f994f7532d52897efbcd1b81a9b710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.license=GPLv2) 2024-03-24T12:17:09.923 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:09 smithi119 podman[79483]: 2024-03-24 12:17:09.83198884 +0000 UTC m=+0.342600392 container init 4217b809fbcab50cb8e67b0758fd9cb99f994f7532d52897efbcd1b81a9b710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T12:17:09.923 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:09 smithi119 podman[79483]: 2024-03-24 12:17:09.837353371 +0000 UTC m=+0.347964912 container start 4217b809fbcab50cb8e67b0758fd9cb99f994f7532d52897efbcd1b81a9b710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T12:17:10.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:09 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:09.923+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:10.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:09 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:09.923+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:10.303 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:09 smithi119 podman[79483]: 2024-03-24 12:17:09.941659992 +0000 UTC m=+0.452271550 container attach 4217b809fbcab50cb8e67b0758fd9cb99f994f7532d52897efbcd1b81a9b710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T12:17:10.303 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 24 12:17:10 smithi119 podman[79483]: 2024-03-24 12:17:10.128810895 +0000 UTC m=+0.639422433 container died 4217b809fbcab50cb8e67b0758fd9cb99f994f7532d52897efbcd1b81a9b710a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T12:17:10.703 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:10 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:10.367+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:10.704 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:10 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:10.367+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:10.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T12:17:10.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:10.862 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T12:17:10.862 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T12:17:10.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:17:10.980 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:10 smithi119 systemd[1]: Stopping Ceph osd.5 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:10.980 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:10 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:10.704+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:10.980 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:10 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:10.704+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:11.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:10 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:10.925+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:11.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:10 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:10.925+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:11.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:11.190+0000 7fe364eff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:11.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:11.190+0000 7fe364eff700 -1 osd.5 842 *** Got signal Terminated *** 2024-03-24T12:17:11.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:11.190+0000 7fe364eff700 -1 osd.5 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:11.663 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:11.320+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:11.663 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:11.320+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:11.973 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:11.664+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:11.974 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:11.664+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:12.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:11.975+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:12.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:11 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:11.975+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:12.700 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:12.345+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:12.700 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:12.345+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:12.971 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:12.701+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:12.971 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:12.701+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:13.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:12.972+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:13.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:12 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:12.972+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:13.698 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:13.380+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:13.698 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:13.380+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:14.014 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:13.699+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:14.014 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:13 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:13.699+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:14.302 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:14 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:14.014+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:14.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:14 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:14.014+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:14.736 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:14 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:14.356+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:14.737 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:14 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:14.356+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:14.999 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:14 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:14.737+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:14.999 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:14 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:14.737+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:15.302 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:14.999+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:15.303 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:14.999+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:15.756 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:15.335+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:15.756 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:15.335+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:16.053 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:15.996+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:36.007842+0000 front 2024-03-24T12:16:36.007801+0000 (oldest deadline 2024-03-24T12:16:59.507655+0000) 2024-03-24T12:17:16.053 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:15.996+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.208835+0000 front 2024-03-24T12:16:44.208845+0000 (oldest deadline 2024-03-24T12:17:07.708676+0000) 2024-03-24T12:17:16.053 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5[45404]: 2024-03-24T12:17:15.996+0000 7fe3614f3700 -1 osd.5 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:50.609642+0000 front 2024-03-24T12:16:50.609597+0000 (oldest deadline 2024-03-24T12:17:15.309524+0000) 2024-03-24T12:17:16.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:15.756+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:16.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:15.756+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:16.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:15 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:15.756+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:16.552 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:16.289+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:16.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:16.289+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:16.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:16.289+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:16.954 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:16 smithi119 podman[79614]: 2024-03-24 12:17:16.668670972 +0000 UTC m=+5.584002982 container died 7e220d89c7ab1dc7fad9915cc0d60440382be3926729f0e1222a4b124894dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:16.955 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:16.768+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:16.955 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:16.768+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:16.955 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:16.768+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:16.955 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:16.647Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-03-24T12:17:16.955 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:16.650Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-03-24T12:17:16.955 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:16.651Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-03-24T12:17:16.955 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:16 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:16.651Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-03-24T12:17:17.271 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 podman[79614]: 2024-03-24 12:17:17.079680836 +0000 UTC m=+5.995012845 container cleanup 7e220d89c7ab1dc7fad9915cc0d60440382be3926729f0e1222a4b124894dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T12:17:17.271 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 bash[79614]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5 2024-03-24T12:17:17.552 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 podman[79635]: 2024-03-24 12:17:17.272574921 +0000 UTC m=+0.607717542 container remove 7e220d89c7ab1dc7fad9915cc0d60440382be3926729f0e1222a4b124894dfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T12:17:17.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:17 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:17.299+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:17.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:17 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:17.299+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:17.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:17 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:17.299+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:17.991 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:17 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:17.804+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:17.991 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:17 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:17.804+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:17.991 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:17 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:17.804+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:17.991 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 podman[79732]: 2024-03-24 12:17:17.593984947 +0000 UTC m=+0.017944496 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:17.991 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 podman[79732]: 2024-03-24 12:17:17.731913406 +0000 UTC m=+0.155873001 container create d9625ce148dddfdb03f04832fffdfef27e2bcee9b364a95f8a2814f4f9e16147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:17.991 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 podman[79732]: 2024-03-24 12:17:17.925965114 +0000 UTC m=+0.349924659 container init d9625ce148dddfdb03f04832fffdfef27e2bcee9b364a95f8a2814f4f9e16147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-24T12:17:17.991 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 podman[79732]: 2024-03-24 12:17:17.931151275 +0000 UTC m=+0.355110809 container start d9625ce148dddfdb03f04832fffdfef27e2bcee9b364a95f8a2814f4f9e16147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:17:17.992 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:17 smithi119 podman[79732]: 2024-03-24 12:17:17.992429278 +0000 UTC m=+0.416388841 container attach d9625ce148dddfdb03f04832fffdfef27e2bcee9b364a95f8a2814f4f9e16147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T12:17:18.256 INFO:journalctl@ceph.osd.5.smithi119.stdout:Mar 24 12:17:18 smithi119 podman[79732]: 2024-03-24 12:17:18.214470299 +0000 UTC m=+0.638429838 container died d9625ce148dddfdb03f04832fffdfef27e2bcee9b364a95f8a2814f4f9e16147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T12:17:18.527 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:18 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:18.257+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:18.527 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:18 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:18.257+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:18.527 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:18 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:18.257+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:18.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T12:17:18.869 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:18.870 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T12:17:18.870 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T12:17:18.870 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.6 2024-03-24T12:17:19.052 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:18 smithi119 systemd[1]: Stopping Ceph osd.6 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:19.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:18 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:18.802+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:19.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:18 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:18.802+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:19.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:18 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:18.802+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:19.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:19.200+0000 7f66dbf63700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:19.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:19.200+0000 7f66dbf63700 -1 osd.6 842 *** Got signal Terminated *** 2024-03-24T12:17:19.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:19.200+0000 7f66dbf63700 -1 osd.6 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:19.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:19.286+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:19.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:19.286+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:19.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:19.286+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:20.289 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:19.849+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:20.289 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:19.849+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:20.289 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:19 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:19.849+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:20.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:20.290+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:20.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:20.290+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:20.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:20.290+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:21.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:20.802+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:21.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:20.802+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:21.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:20 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:20.802+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:21.762 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:21.330+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:21.762 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:21.330+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:21.762 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:21.330+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:22.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:21.762+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:22.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:21.762+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:22.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:21.762+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:22.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:21 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:21.762+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:22.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:22.282+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:22.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:22.282+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:22.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:22.282+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:23.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:22.719+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:23.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:22.719+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:23.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:22.719+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:23.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:22 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:22.719+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:23.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:23.265+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:35.786322+0000 front 2024-03-24T12:16:35.786408+0000 (oldest deadline 2024-03-24T12:17:00.486226+0000) 2024-03-24T12:17:23.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:23.265+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:43.987325+0000 front 2024-03-24T12:16:43.987265+0000 (oldest deadline 2024-03-24T12:17:06.287149+0000) 2024-03-24T12:17:23.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:23.265+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:51.389308+0000 front 2024-03-24T12:16:51.389263+0000 (oldest deadline 2024-03-24T12:17:15.489177+0000) 2024-03-24T12:17:23.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6[51486]: 2024-03-24T12:17:23.265+0000 7f66d8557700 -1 osd.6 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.390153+0000 front 2024-03-24T12:16:58.390116+0000 (oldest deadline 2024-03-24T12:17:23.089961+0000) 2024-03-24T12:17:24.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:23.724+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:24.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:23.724+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:24.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:23.724+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:24.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:23 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:23.725+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:24.864 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:24 smithi119 podman[79863]: 2024-03-24 12:17:24.598407832 +0000 UTC m=+5.502352355 container died 3846a6b900dae3e1d1189e8c57e86044d65d0f8165b63e78f5f3c1a1e7f432b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:24.864 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:24 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:24.773+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:24.865 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:24 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:24.773+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:24.865 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:24 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:24.773+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:24.865 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:24 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:24.773+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:25.256 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:24 smithi119 podman[79863]: 2024-03-24 12:17:24.964992088 +0000 UTC m=+5.868936607 container cleanup 3846a6b900dae3e1d1189e8c57e86044d65d0f8165b63e78f5f3c1a1e7f432b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:25.256 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:24 smithi119 bash[79863]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6 2024-03-24T12:17:25.553 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:25 smithi119 podman[79887]: 2024-03-24 12:17:25.25807775 +0000 UTC m=+0.657708463 container remove 3846a6b900dae3e1d1189e8c57e86044d65d0f8165b63e78f5f3c1a1e7f432b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:25.860 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:25 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:25.769+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:25.860 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:25 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:25.769+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:25.860 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:25 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:25.769+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:25.860 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:25 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:25.769+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.413207+0000 front 2024-03-24T12:16:58.413226+0000 (oldest deadline 2024-03-24T12:17:24.813926+0000) 2024-03-24T12:17:25.860 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:25 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:25.769+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:25.860 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:25 smithi119 podman[79987]: 2024-03-24 12:17:25.492435798 +0000 UTC m=+0.017411254 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:25.860 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:25 smithi119 podman[79987]: 2024-03-24 12:17:25.633884565 +0000 UTC m=+0.158860024 container create ea3f35a604a3c1e2989e7a03cf3ae2b38961d8b55ed09cf178db02803679b37d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T12:17:25.861 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:25 smithi119 podman[79987]: 2024-03-24 12:17:25.861377106 +0000 UTC m=+0.386352565 container init ea3f35a604a3c1e2989e7a03cf3ae2b38961d8b55ed09cf178db02803679b37d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T12:17:26.152 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:25 smithi119 podman[79987]: 2024-03-24 12:17:25.867074101 +0000 UTC m=+0.392049554 container start ea3f35a604a3c1e2989e7a03cf3ae2b38961d8b55ed09cf178db02803679b37d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T12:17:26.153 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:25 smithi119 podman[79987]: 2024-03-24 12:17:25.961124122 +0000 UTC m=+0.486099589 container attach ea3f35a604a3c1e2989e7a03cf3ae2b38961d8b55ed09cf178db02803679b37d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T12:17:26.513 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 24 12:17:26 smithi119 podman[79987]: 2024-03-24 12:17:26.15453852 +0000 UTC m=+0.679513971 container died ea3f35a604a3c1e2989e7a03cf3ae2b38961d8b55ed09cf178db02803679b37d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:26.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T12:17:26.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:26.868 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T12:17:26.868 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T12:17:26.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.7 2024-03-24T12:17:27.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:26 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:26.807+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:27.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:26 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:26.807+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:27.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:26 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:26.807+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:27.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:26 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:26.807+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.413207+0000 front 2024-03-24T12:16:58.413226+0000 (oldest deadline 2024-03-24T12:17:24.813926+0000) 2024-03-24T12:17:27.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:26 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:26.807+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:27.054 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:26 smithi119 systemd[1]: Stopping Ceph osd.7 for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:27.552 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.219+0000 7fe386955700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:27.553 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.219+0000 7fe386955700 -1 osd.7 842 *** Got signal Terminated *** 2024-03-24T12:17:27.553 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.219+0000 7fe386955700 -1 osd.7 842 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:17:28.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.767+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:28.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.767+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:28.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.767+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:28.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.767+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.413207+0000 front 2024-03-24T12:16:58.413226+0000 (oldest deadline 2024-03-24T12:17:24.813926+0000) 2024-03-24T12:17:28.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:27 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:27.767+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:29.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:28 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:28.765+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:29.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:28 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:28.765+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:29.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:28 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:28.765+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:29.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:28 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:28.765+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.413207+0000 front 2024-03-24T12:16:58.413226+0000 (oldest deadline 2024-03-24T12:17:24.813926+0000) 2024-03-24T12:17:29.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:28 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:28.765+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:30.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:29 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:29.799+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:30.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:29 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:29.799+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:30.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:29 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:29.799+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:30.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:29 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:29.799+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.413207+0000 front 2024-03-24T12:16:58.413226+0000 (oldest deadline 2024-03-24T12:17:24.813926+0000) 2024-03-24T12:17:30.053 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:29 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:29.799+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:31.302 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:30 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:30.840+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:31.303 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:30 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:30.840+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:31.303 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:30 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:30.840+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:31.303 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:30 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:30.840+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.413207+0000 front 2024-03-24T12:16:58.413226+0000 (oldest deadline 2024-03-24T12:17:24.813926+0000) 2024-03-24T12:17:31.303 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:30 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:30.840+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:32.204 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:31.816+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6803 osd.0 since back 2024-03-24T12:16:34.807862+0000 front 2024-03-24T12:16:34.807746+0000 (oldest deadline 2024-03-24T12:16:58.307718+0000) 2024-03-24T12:17:32.204 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:31.816+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6807 osd.1 since back 2024-03-24T12:16:44.609994+0000 front 2024-03-24T12:16:44.609945+0000 (oldest deadline 2024-03-24T12:17:06.309873+0000) 2024-03-24T12:17:32.205 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:31.816+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6811 osd.2 since back 2024-03-24T12:16:49.211313+0000 front 2024-03-24T12:16:49.211442+0000 (oldest deadline 2024-03-24T12:17:15.111185+0000) 2024-03-24T12:17:32.205 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:31.816+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.114:6815 osd.3 since back 2024-03-24T12:16:58.413207+0000 front 2024-03-24T12:16:58.413226+0000 (oldest deadline 2024-03-24T12:17:24.813926+0000) 2024-03-24T12:17:32.205 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:31.816+0000 7fe382f49700 -1 osd.7 842 heartbeat_check: no reply from 172.21.15.119:6802 osd.4 since back 2024-03-24T12:17:07.115017+0000 front 2024-03-24T12:17:07.114981+0000 (oldest deadline 2024-03-24T12:17:31.214985+0000) 2024-03-24T12:17:32.205 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:31 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7[57555]: 2024-03-24T12:17:31.816+0000 7fe382f49700 -1 osd.7 842 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.34433.0:2234 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e842) 2024-03-24T12:17:32.803 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:32 smithi119 podman[80120]: 2024-03-24 12:17:32.699223418 +0000 UTC m=+5.608011087 container died 65db3ff508e3bcec5ddce46b687866aa56c876c3b2a05b5e0288d61b4e2ee3dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7, RELEASE=squid-db0330b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:17:33.302 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:33 smithi119 podman[80120]: 2024-03-24 12:17:33.083364755 +0000 UTC m=+5.992152432 container cleanup 65db3ff508e3bcec5ddce46b687866aa56c876c3b2a05b5e0288d61b4e2ee3dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T12:17:33.303 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:33 smithi119 bash[80120]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7 2024-03-24T12:17:33.691 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:33 smithi119 podman[80151]: 2024-03-24 12:17:33.334359614 +0000 UTC m=+0.625591595 container remove 65db3ff508e3bcec5ddce46b687866aa56c876c3b2a05b5e0288d61b4e2ee3dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T12:17:34.010 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:33 smithi119 podman[80254]: 2024-03-24 12:17:33.595534731 +0000 UTC m=+0.018722000 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:17:34.011 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:33 smithi119 podman[80254]: 2024-03-24 12:17:33.710217933 +0000 UTC m=+0.133405180 container create 1edecff2ad0b59d93e7c805650e69ec3a3c37a269f61be49e2be029e68f3dfc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T12:17:34.011 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:33 smithi119 podman[80254]: 2024-03-24 12:17:33.920698722 +0000 UTC m=+0.343885975 container init 1edecff2ad0b59d93e7c805650e69ec3a3c37a269f61be49e2be029e68f3dfc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T12:17:34.011 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:33 smithi119 podman[80254]: 2024-03-24 12:17:33.926112822 +0000 UTC m=+0.349300077 container start 1edecff2ad0b59d93e7c805650e69ec3a3c37a269f61be49e2be029e68f3dfc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=squid-db0330b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T12:17:34.011 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:34 smithi119 podman[80254]: 2024-03-24 12:17:34.012161361 +0000 UTC m=+0.435348617 container attach 1edecff2ad0b59d93e7c805650e69ec3a3c37a269f61be49e2be029e68f3dfc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD) 2024-03-24T12:17:34.303 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:34 smithi119 podman[80254]: 2024-03-24 12:17:34.216581762 +0000 UTC m=+0.639769013 container died 1edecff2ad0b59d93e7c805650e69ec3a3c37a269f61be49e2be029e68f3dfc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T12:17:34.771 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 24 12:17:34 smithi119 podman[80300]: 2024-03-24 12:17:34.756883565 +0000 UTC m=+0.530905259 container remove 1edecff2ad0b59d93e7c805650e69ec3a3c37a269f61be49e2be029e68f3dfc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T12:17:34.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T12:17:34.815 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:34.816 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T12:17:34.816 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-24T12:17:34.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-24T12:17:35.237 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 12:17:34 smithi114 systemd[1]: Stopping Ceph rgw.foo.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:35.238 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 12:17:35 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a[70500]: 2024-03-24T12:17:35.094+0000 7fa8ef367700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:17:35.238 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 12:17:35 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a[70500]: 2024-03-24T12:17:35.094+0000 7fa8fdbd7ac0 -1 shutting down 2024-03-24T12:17:45.373 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 12:17:45 smithi114 bash[162144]: time="2024-03-24T12:17:45Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-24T12:17:45.373 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 12:17:45 smithi114 podman[162144]: 2024-03-24 12:17:45.122700232 +0000 UTC m=+10.111711811 container died 669e0c06dcb5abbfc02a48c1ee9bdbee3ae1053003f29a3cead2c68da528bc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T12:17:45.738 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 12:17:45 smithi114 podman[162144]: 2024-03-24 12:17:45.482177456 +0000 UTC m=+10.471189072 container cleanup 669e0c06dcb5abbfc02a48c1ee9bdbee3ae1053003f29a3cead2c68da528bc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T12:17:45.738 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Mar 24 12:17:45 smithi114 bash[162144]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-24T12:17:45.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-24T12:17:45.938 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:45.938 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-24T12:17:45.938 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T12:17:45.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-412c996a-e9d2-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T12:17:46.235 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:45 smithi119 systemd[1]: Stopping Ceph prometheus.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:46.516 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:17:46.516 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.236Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.237Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.240Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.240Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a[67831]: ts=2024-03-24T12:17:46.240Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T12:17:46.517 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 podman[80385]: 2024-03-24 12:17:46.247292686 +0000 UTC m=+0.117196672 container died e0c807da30bec18da9f5559849e0267746eb32b512233ababc38c9405c103c76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:46.803 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 podman[80385]: 2024-03-24 12:17:46.625180158 +0000 UTC m=+0.495084179 container cleanup e0c807da30bec18da9f5559849e0267746eb32b512233ababc38c9405c103c76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:46.803 INFO:journalctl@ceph.prometheus.a.smithi119.stdout:Mar 24 12:17:46 smithi119 bash[80385]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T12:17:47.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412c996a-e9d2-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T12:17:47.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:17:47.080 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T12:17:47.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm rm-cluster --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:17:47.268 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T12:17:49.131 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:48 smithi114 systemd[1]: Stopping Ceph alertmanager.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:49.132 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:49 smithi114 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a[81518]: ts=2024-03-24T12:17:49.081Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:17:49.452 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:49 smithi114 podman[162665]: 2024-03-24 12:17:49.132872283 +0000 UTC m=+0.154775102 container died 64dc685b8a6da1024b22b31701928592f12e3932a56608b37732f988b75083ae (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:49.452 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:49 smithi114 podman[162665]: 2024-03-24 12:17:49.453668529 +0000 UTC m=+0.475571346 container cleanup 64dc685b8a6da1024b22b31701928592f12e3932a56608b37732f988b75083ae (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:49.737 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:49 smithi114 bash[162665]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T12:17:49.737 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:49 smithi114 podman[162684]: 2024-03-24 12:17:49.654773165 +0000 UTC m=+0.522068622 container remove 64dc685b8a6da1024b22b31701928592f12e3932a56608b37732f988b75083ae (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:49.737 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:49 smithi114 podman[162684]: 2024-03-24 12:17:49.738680269 +0000 UTC m=+0.605975743 volume remove d659fe8b7130334d54b9c67148bfe2360ee02078e7339d477f093875ffe04ec6 2024-03-24T12:17:50.453 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:50 smithi114 systemd[1]: Stopping Ceph node-exporter.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:50.453 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:50 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T12:17:50.453 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:50 smithi114 systemd[1]: Stopped Ceph alertmanager.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:17:50.453 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Mar 24 12:17:50 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 2.449s CPU time. 2024-03-24T12:17:50.737 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:50 smithi114 podman[162800]: 2024-03-24 12:17:50.45500906 +0000 UTC m=+0.109467236 container died 13608fac3d0d4cb64423af21d8313fe985db35a2d0ea30a1c9b7c8f7e8a033e8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:51.176 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:50 smithi114 podman[162800]: 2024-03-24 12:17:50.782978696 +0000 UTC m=+0.437436871 container cleanup 13608fac3d0d4cb64423af21d8313fe985db35a2d0ea30a1c9b7c8f7e8a033e8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:51.176 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:50 smithi114 bash[162800]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T12:17:51.176 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:50 smithi114 podman[162819]: 2024-03-24 12:17:50.975744529 +0000 UTC m=+0.522060467 container remove 13608fac3d0d4cb64423af21d8313fe985db35a2d0ea30a1c9b7c8f7e8a033e8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:17:51.176 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:50 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:17:51.487 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:51 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T12:17:51.487 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:51 smithi114 systemd[1]: Stopped Ceph node-exporter.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:17:51.487 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Mar 24 12:17:51 smithi114 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 5.229s CPU time. 2024-03-24T12:17:51.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm rm-cluster --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:17:51.992 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T12:17:53.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:17:53 smithi119 systemd[1]: Stopping Ceph iscsi.iscsi.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:17:53.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:17:53 smithi119 ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a[62412]: debug Shutdown received 2024-03-24T12:18:03.767 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:03 smithi119 bash[80806]: time="2024-03-24T12:18:03Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-24T12:18:03.767 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:03 smithi119 podman[80806]: 2024-03-24 12:18:03.488529152 +0000 UTC m=+10.125471811 container died 9f94706c27741e452ce286ae70f7732a5403286b2c3e5471ef53680b7a14abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T12:18:04.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:03 smithi119 podman[80806]: 2024-03-24 12:18:03.86749513 +0000 UTC m=+10.504437769 container cleanup 9f94706c27741e452ce286ae70f7732a5403286b2c3e5471ef53680b7a14abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T12:18:04.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:03 smithi119 bash[80806]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a 2024-03-24T12:18:04.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:04 smithi119 podman[80831]: 2024-03-24 12:18:04.12723123 +0000 UTC m=+0.636157202 container remove 9f94706c27741e452ce286ae70f7732a5403286b2c3e5471ef53680b7a14abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:18:04.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:04 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T12:18:04.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:04 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-24T12:18:04.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:04 smithi119 systemd[1]: Stopped Ceph iscsi.iscsi.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:18:04.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi119.stdout:Mar 24 12:18:04 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Consumed 4.341s CPU time. 2024-03-24T12:18:05.802 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:05 smithi119 systemd[1]: Stopping Ceph grafana.a for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:18:06.207 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:05 smithi119 podman[81119]: 2024-03-24 12:18:05.9304255 +0000 UTC m=+0.120080141 container died c52fd27442fb885917fe2dd72f1fe9eb2623b7c68f63324e2eab8281a1d2746d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a, release=239.1651231664, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, version=8.5, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, description=Ceph Grafana Container, distribution-scope=public, name=ubi8, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T12:18:06.497 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:06 smithi119 podman[81119]: 2024-03-24 12:18:06.29137409 +0000 UTC m=+0.481028733 container cleanup c52fd27442fb885917fe2dd72f1fe9eb2623b7c68f63324e2eab8281a1d2746d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, io.openshift.tags=base rhel8, distribution-scope=public, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T12:18:06.498 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:06 smithi119 bash[81119]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a 2024-03-24T12:18:06.803 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:06 smithi119 podman[81138]: 2024-03-24 12:18:06.542744952 +0000 UTC m=+0.610799863 container remove c52fd27442fb885917fe2dd72f1fe9eb2623b7c68f63324e2eab8281a1d2746d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, architecture=x86_64, io.openshift.expose-services=, io.openshift.tags=base rhel8, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897) 2024-03-24T12:18:06.803 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:06 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:18:06.803 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:06 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-24T12:18:06.803 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:06 smithi119 systemd[1]: Stopped Ceph grafana.a for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:18:06.803 INFO:journalctl@ceph.grafana.a.smithi119.stdout:Mar 24 12:18:06 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@grafana.a.service: Consumed 18.469s CPU time. 2024-03-24T12:18:07.302 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:06 smithi119 systemd[1]: Stopping Ceph node-exporter.b for 412c996a-e9d2-11ee-95cd-87774f69a715... 2024-03-24T12:18:07.303 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 podman[81259]: 2024-03-24 12:18:07.142074526 +0000 UTC m=+0.096616913 container died dc2d3e8ddd15b81d19a81b64ac15ae1f97ed60647b1caf519c505204870625ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:18:07.669 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 podman[81259]: 2024-03-24 12:18:07.436997776 +0000 UTC m=+0.391540164 container cleanup dc2d3e8ddd15b81d19a81b64ac15ae1f97ed60647b1caf519c505204870625ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:18:07.670 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 bash[81259]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T12:18:07.937 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 podman[81278]: 2024-03-24 12:18:07.671305856 +0000 UTC m=+0.530562447 container remove dc2d3e8ddd15b81d19a81b64ac15ae1f97ed60647b1caf519c505204870625ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-412c996a-e9d2-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:18:07.937 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:18:07.937 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T12:18:07.938 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 systemd[1]: Stopped Ceph node-exporter.b for 412c996a-e9d2-11ee-95cd-87774f69a715. 2024-03-24T12:18:07.938 INFO:journalctl@ceph.node-exporter.b.smithi119.stdout:Mar 24 12:18:07 smithi119 systemd[1]: ceph-412c996a-e9d2-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 5.219s CPU time. 2024-03-24T12:18:08.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:18:08.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:18:08.484 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T12:18:08.486 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924/remote/smithi114/crash 2024-03-24T12:18:08.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:18:08.539 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:18:08.539 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:18:08.541 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924/remote/smithi119/crash 2024-03-24T12:18:08.542 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:18:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:18:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:18:08.577 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T12:18:08.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-24T12:18:08.615 INFO:teuthology.orchestra.run.smithi114.stdout:2024-03-24T11:50:00.000145+0000 mon.a (mon.0) 2677 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-24T12:18:08.615 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T12:18:08.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[SEC\]' /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-24T12:18:08.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]' /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-24T12:18:08.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[WRN\]' /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-24T12:18:08.832 INFO:teuthology.orchestra.run.smithi114.stdout:2024-03-24T11:50:00.000145+0000 mon.a (mon.0) 2677 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-24T12:18:08.833 INFO:tasks.cephadm:Compressing logs... 2024-03-24T12:18:08.833 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:18:08.875 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:18:08.899 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T12:18:08.899 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-03-24T12:18:08.899 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T12:18:08.899 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:18:08.901 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.a.log: -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-24T12:18:08.901 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T12:18:08.901 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T12:18:08.901 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T12:18:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T12:18:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:: No such file or directory 2024-03-24T12:18:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T12:18:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:18:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T12:18:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T12:18:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T12:18:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T12:18:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-24T12:18:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:18:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/tcmu-runner.log: 67.3% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-24T12:18:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:18:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.cephadm.log: 81.7% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:18:08.906 INFO:teuthology.orchestra.run.smithi119.stderr: 85.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:18:08.908 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:18:08.908 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T12:18:08.908 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T12:18:08.909 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mgr.y.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:18:08.909 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:18:08.912 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-volume.log: 88.1% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:18:08.913 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.b.log: 92.3% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:18:08.915 INFO:teuthology.orchestra.run.smithi119.stderr: 92.4% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T12:18:08.919 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:18:08.920 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.cephadm.log: 95.3% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:18:08.921 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:18:08.921 INFO:teuthology.orchestra.run.smithi119.stderr: 90.8% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:18:08.933 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T12:18:08.942 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log: 93.4% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:18:08.946 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:18:08.959 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T12:18:08.967 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-volume.log: 93.5% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-24T12:18:08.989 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:18:09.451 INFO:teuthology.orchestra.run.smithi114.stderr: 90.1% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T12:18:10.347 INFO:teuthology.orchestra.run.smithi119.stderr: 92.4% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T12:18:10.538 INFO:teuthology.orchestra.run.smithi114.stderr: 92.2% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T12:18:12.401 INFO:teuthology.orchestra.run.smithi114.stderr: 91.5% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T12:18:17.371 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T12:18:17.912 INFO:teuthology.orchestra.run.smithi119.stderr: 93.9% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T12:18:18.195 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T12:18:18.238 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T12:18:18.315 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T12:18:18.459 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T12:18:18.475 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T12:18:18.478 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-24T12:18:18.478 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m9.591s 2024-03-24T12:18:18.478 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m41.854s 2024-03-24T12:18:18.478 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m1.506s 2024-03-24T12:18:19.519 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/412c996a-e9d2-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T12:18:19.521 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-24T12:18:19.521 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m10.632s 2024-03-24T12:18:19.521 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m37.605s 2024-03-24T12:18:19.521 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m1.345s 2024-03-24T12:18:19.521 INFO:tasks.cephadm:Archiving logs... 2024-03-24T12:18:19.522 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924/remote/smithi114/log 2024-03-24T12:18:19.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:18:25.070 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924/remote/smithi119/log 2024-03-24T12:18:25.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:18:30.289 INFO:tasks.cephadm:Removing cluster... 2024-03-24T12:18:30.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo cephadm rm-cluster --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 --force 2024-03-24T12:18:30.482 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T12:18:30.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm rm-cluster --fsid 412c996a-e9d2-11ee-95cd-87774f69a715 --force 2024-03-24T12:18:30.975 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: 412c996a-e9d2-11ee-95cd-87774f69a715 2024-03-24T12:18:31.278 INFO:tasks.cephadm:Teardown complete 2024-03-24T12:18:31.278 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T12:18:31.288 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T12:18:31.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T12:18:31.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T12:18:31.365 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T12:18:31.411 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T12:18:31.411 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:18:31.413 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:18:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:18:31.427 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:18:31.430 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:18:31.430 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-24T12:18:31.431 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +800us[ +800us] +/- 68ms 2024-03-24T12:18:31.431 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 +13us[ +11us] +/- 49ms 2024-03-24T12:18:31.431 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -858us[ -858us] +/- 73ms 2024-03-24T12:18:31.431 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:18:31.432 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:18:31.432 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-24T12:18:31.432 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +816us[ +819us] +/- 68ms 2024-03-24T12:18:31.433 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 +18us[ +21us] +/- 49ms 2024-03-24T12:18:31.433 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -868us[ -865us] +/- 73ms 2024-03-24T12:18:31.433 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:18:31.433 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T12:18:31.474 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T12:18:31.475 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T12:18:31.491 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:18:31.574 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:18:31.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:18:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:18:31.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:18:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:18:31.712 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T12:18:31.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T12:18:31.730 INFO:teuthology.task.internal:Duration was 3135.938873 seconds 2024-03-24T12:18:31.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T12:18:31.741 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T12:18:31.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:18:31.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:18:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:18:31.795 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:18:32.090 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T12:18:32.090 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-03-24T12:18:32.091 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:18:32.110 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-03-24T12:18:32.110 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:18:32.152 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T12:18:32.152 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:18:32.155 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:18:32.221 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T12:18:32.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:18:32.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:18:32.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T12:18:32.681 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T12:18:32.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:18:32.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:18:32.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T12:18:32.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:18:32.750 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:18:32.773 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-03-24T12:18:32.800 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-03-24T12:18:32.828 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:18:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:18:32.842 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:18:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:18:32.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T12:18:32.893 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T12:18:32.894 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924/remote/smithi114 2024-03-24T12:18:32.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:18:32.961 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924/remote/smithi119 2024-03-24T12:18:32.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:18:33.001 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T12:18:33.001 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:18:33.003 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:18:33.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T12:18:33.061 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T12:18:33.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T12:18:33.076 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T12:18:33.076 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:18:33.079 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:18:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: 265622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:18 /home/ubuntu/cephtest 2024-03-24T12:18:33.103 INFO:teuthology.orchestra.run.smithi119.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:18 /home/ubuntu/cephtest 2024-03-24T12:18:33.104 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T12:18:33.115 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T12:18:33.138 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T12:18:33.164 DEBUG:teuthology.nuke:shortname: smithi114 2024-03-24T12:18:33.164 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:18:33.187 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:17:24.167511', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAxsFIyqivlXUt6ACFkD4M+bW95KpstEbHQuXEvKg3MhB77DoW1P4FekH2ME4GHxcgvWaLTnIjzwhOkME/jJkno='} 2024-03-24T12:18:33.188 DEBUG:teuthology.nuke:shortname: smithi119 2024-03-24T12:18:33.188 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:18:33.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:17:24.165868', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNY7Xz0JLlpCSUPwvmjpOZDtcny0WnTS/9xEOHSFotE3picEqPPWYAHn12VzT7Twb0OuHpWcLqfx3L4AoCjHWM4='} 2024-03-24T12:18:33.242 INFO:teuthology.orchestra.console.smithi114:Power off 2024-03-24T12:18:33.242 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:18:33.258 INFO:teuthology.orchestra.console.smithi119:Power off 2024-03-24T12:18:33.258 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:18:33.266 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-03-24T12:18:33.266 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:33.279 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-03-24T12:18:33.280 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:33.290 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-24T12:18:33.301 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-24T12:18:37.291 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:37.302 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:37.401 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-24T12:18:37.411 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-24T12:18:41.406 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:41.416 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:41.519 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-24T12:18:41.525 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-24T12:18:45.525 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:45.533 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:18:45.636 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-03-24T12:18:45.643 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-03-24T12:18:45.737 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-03-24T12:18:45.744 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-03-24T12:18:45.848 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} duration: 3135.938873052597 failure_reason: '"2024-03-24T11:50:00.000145+0000 mon.a (mon.0) 2677 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-24T12:18:45.849 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T12:18:45.929 INFO:teuthology.run:FAIL